skorotkiewicz/rDrama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

post scheduling (#554)

Browse source 
* prepare codebase to create scheduled tasks

there is some prep work involved with this. the scheduler would be happy
 if this work was done. simply, we extract out the `created_utc`
 interface from *everything* that uses it such that we don't have to
 repeat ourselves a bunch. all fun stuff.

next commit is the meat of it.

* cron: basic backend work for scheduler

* avoid ipmort loop

* attempt 2 at fixing import loops

* parathensize because operator precedence

* delete file that came back for some reason.

* does NOPing the oauth apps work?

* import late and undo clients.py change

* stringify column names.

* reorder imports.

* remove task reference

* fix missing mapper object

* make coupled to repeatabletask i guess

* sanitize: fix sanitize imports

* import shadowing crap

* re-shadow shadowed variable

* fix regexes

* use the correct not operator

* readd missing commit

* scheduler: SQLA only allows concrete relations

* implement submission scheduler

* fix import loop with db_session

* get rid of import loop in submission.py and comment.py

* remove import loops by deferring import until function clal

* i give up.

* awful.

* ...

* fix another app import loop

* fix missing import in route handler

* fix import error in wrappers.py

* fix wrapper error

* call update wrapper in the admin_level_required case

* :marseyshrug:

* fix issue with wrapper

* some cleanup and some fixes

* some more cleanup

let's avoid polluting scopes where we can.

* ...

* add SCHEDULED_POSTS permission.

* move const.py into config like the other files.

* style fixes.

* lock table for concurrency improvements

* don't attempt to commit on errors

* Refactor code, create `TaskRunContext`, create python callable task type.

* use import contextlib

* testing stuff i guess.

* handle repeatable tasks properly.

* Attempt another fix at fighting the mapper

* do it right ig

* SQLA1.4 doesn't support nested polymorphism ig

* fix errenous class import

* fix mapper errors

* import app in wrappers.py

* fix import failures and stuff like that.

* embed and import fixes

* minor formatting changes.

* Add running state enum and don't attempt to check for currently running tasks.

* isort

* documentation, style, and commit after each task.

* Add completion time and more docs, rename, etc

* document `CRON_SLEEP_SECONDS` better.

* add note about making LiteralString

* filter out tasks that have been run in the future

* reference RepeatableTask's `__tablename__` directly

* use a master/slave configuration for tasks

the master periodically checks to see if the slave is alive, healthy,
and not taking too many resources, and if applicable kills its
child and restarts it.

only one relation is supported at the moment.

* don't duplicate process unnecessarily

* note impl detail, add comments

* fix imports.

* getting imports to stop being stupid.

* environment notes.

* syntax derp

* *sigh*

* stupid environment stuff

* add UI for submitting a scheduled post

* stupid things i need to fix the user class

* ...

* fix template

* add formkey

* pass v

* add hour and minute field

* bleh

* remove concrete

* the sqlalchemy docs are wrong

* fix me being dumb and not understanding error messages

* missing author attribute for display

* author_name property

* it's a property

* with_polymorphic i think fixes this

* dsfavgnhmjk

* *sigh*

* okay try this again

* try getting rid of the comment section

* include -> extends

* put the div outside of the thing.

* fix user page listings :/

* mhm

* i hate this why isn't this working

* this should fix it

* Fix posts being set as disabled by default

* form UI imrpovements

* label

* <textarea>s should have their closing tag

* UI fixes.

* and fix errenous spinner thing.

* don't abort(415) when browsers send 0 length files for some reason

* UI improvements

* line break.

* CSS :S

* better explainer

* don't show moderation buttons for scheduled posts

* ...

* meh

* add edit form

* include forms on default page.

* fix hour minute selectino.

* improve ui i guess and add api

* Show previous postings on scheduled task page

* create task id

* sqla

* posts -> submissions

* fix OTM relationship

* edit URL

* use common formkey control

* Idk why this isn't working

* Revert "Idk why this isn't working"

This reverts commit 3b93f741df.

* does removing viewonly fix it?

* don't import routes on db migrations

* apparently this has to be a string

* UI improvements redux

* margins and stuff

* add cron to supervisord

* remove stupid duplication

* typo fix

* postgres syntax error

* better lock and error handling

* add relationship between task and runs

* fix some ui stuff

* fix incorrect timestamp comparison

* ...

* Fix logic errors blocking scheduled posts

Two bugs here:
  - RepeatableTask.run_time_last <= now: run_time_last is NULL by
    default. NULL is not greater than, less than, or equal to any
    value. We use NULL to signify a never-run task; check for that
    condition when building the task list.
  - `6 <= weekday <= 0`: there is no integer that is both gte 6 and
    lte 0. This was always false.

* pasthrough worker process STDOUT and STDERR

* Add scheduler to admin panel

* scheduler

* fix listing and admin home

* date formatting ixes

* fix ages

* task user interface

* fix some more import crap i have to deal with

* fix typing

* avoid import loop

* UI fixes

* fix incorrect type

* task type

* Scheduled task UI improvements (add runs and stuff)

* make the width a lil bit smaller

* task runs.

* fix submit page

* add alembic migration

* log on startup

* Fix showing edit button

* Fix logic for `can_edit` (accidentally did `author_id` instead of `id`)

* Broad review pass

Review:
  - Call `invalidate_cache` with `is_html=` explicitly for clarity,
    rather than a bare boolean in the call args.
  - Remove `marseys_const*` and associated stateful const system:
    the implementation was good if we needed them, but TheMotte
    doesn't use emoji, and a greenfield emoji system would likely
    not keep those darned lists floating in thread-local scope.
    Also they were only needed for goldens and random emoji, which
    are fairly non-central features.
  - Get `os.environ` fully out of the templates by using the new
    constants we already have in files.helpers.config.environment.
  - Given files.routes.posts cleanup,get rid of shop discount dict.
    It's already a mapping of badge IDs to discounts for badges that
    likely won't continue to exist (if they even do at present).
  - RepeatableTaskRun.exception: use `@property.setter` instead of
    overriding `__setattr__`.

Fix:
  - Welcome message literal contained an indented Markdown code block.
  - Condition to show "View source" button changed to show source to
    logged out. This may well be a desirable change, but it's not
    clearly intended here.

* Fix couple of routing issues

* fix 400 with post body editing

* Add error handler for HTTP 415

* fix router giving wrong arg name to handler

* Use supervisord to monitor memory rather than DIY

Also means we're using pip for getting supervisord now, so we don't rely
on the Debian image base for any packages.

* fix task run elapsed time display

* formatting and removing redundant code

* Fix missing ModAction import

* dates and times fixes

* Having to modify imports here anyway, might as
well change it.

* correct documentation.

* don't use urlunparse

* validators: import sanitize instead of from syntax

* cron: prevent races on task running

RepeatableTask.run_state_enum acts as the mutex on repeatable tasks.
Previously, the list of tasks to run was acquired before individually
locking each task. However, there was a period where the table is both
unlocked and the tasks are in state WAITING between those points.
This could potentially have led to two 'cron' processes each running the
same task simultaneously. Instead, we check for runnability both when
building the preliminary list and when mutexing the task via run state
in the database.

Also:
  - g.db and the cron db object are both instances of `Session`, not
    `scoped_session` because they are obtained from
    `scoped_session.__call__`, which acts as a `Session` factory.
    Propagate this to the type hints.
  - Sort order of task run submissions so /tasks/scheduled_posts/<id>
    "Previous Task Runs" listings are useful.

* Notify followers on post publication

This was old behavior lost in the refactoring of the submit endpoint.

Also fix an AttributeError in `Follow.__repr__` which carried over
from all the repr copypasta.

* Fix image attachment

Any check for `file.content_length` relies on browsers sending
Content-Length headers with the request. It seems that few actually do.

The pre-refactor approach was to check for truthiness, which excludes
both None and the strange empty strings that we seem to get in absence
of a file upload. We return to doing so.

---------

Co-authored-by: TLSM <duolsm@outlook.com>
This commit is contained in:
justcool393 2023-03-29 14:32:48 -07:00 • committed by GitHub
parent 9133d35e6f
commit be952c2771
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
121 changed files with 3284 additions and 1808 deletions
Download patch file Download diff file Expand all files Collapse all files

6
Dockerfile
View file

@ -4,7 +4,7 @@ FROM python:3.10 AS base
ARG DEBIAN_FRONTEND=noninteractive
RUN apt update && apt -y upgrade && apt install -y supervisor
RUN apt update && apt -y upgrade
# we'll end up blowing away this directory via docker-compose
WORKDIR /service
@ -25,7 +25,7 @@ ENV FLASK_APP=files/cli:app
FROM base AS release
COPY bootstrap/supervisord.conf.release /etc/supervisord.conf
CMD [ "/usr/bin/supervisord", "-c", "/etc/supervisord.conf" ]
CMD [ "/usr/local/bin/supervisord", "-c", "/etc/supervisord.conf" ]
###################################################################
@ -37,7 +37,7 @@ COPY thirdparty/sqlalchemy-easy-profile sqlalchemy-easy-profile
RUN cd sqlalchemy-easy-profile && python3 setup.py install
COPY bootstrap/supervisord.conf.dev /etc/supervisord.conf
CMD [ "/usr/bin/supervisord", "-c", "/etc/supervisord.conf" ]
CMD [ "/usr/local/bin/supervisord", "-c", "/etc/supervisord.conf" ]
###################################################################

19
bootstrap/supervisord.conf.dev
View file

@ -3,6 +3,13 @@ nodaemon=true
pidfile=/tmp/supervisord.pid
logfile=/tmp/supervisord.log
[unix_http_server]
file = /run/supervisor.sock
# complains about no authentication on startup, but it's 0700 root:root by default
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:service]
directory=/service
command=sh -c 'python3 -m flask db upgrade && WERKZEUG_DEBUG_PIN=off ENABLE_SERVICES=true python3 -m flask --debug run --host=0.0.0.0 --port=80'
@ -10,3 +17,15 @@ stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:cron]
directory=/service
command=sh -c 'python3 -m flask cron'
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[eventlistener:memmon]
command=memmon -p cron=300MB
events=TICK_60

19
bootstrap/supervisord.conf.release
View file

@ -3,6 +3,13 @@ nodaemon=true
pidfile=/tmp/supervisord.pid
logfile=/tmp/supervisord.log
[unix_http_server]
file = /run/supervisor.sock
# complains about no authentication on startup, but it's 0700 root:root by default
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:service]
directory=/service
command=sh -c 'python3 -m flask db upgrade && ENABLE_SERVICES=true gunicorn files.__main__:app -k gevent -w ${CORE_OVERRIDE:-$(( `nproc` * 2 ))} --reload -b 0.0.0.0:80 --max-requests 1000 --max-requests-jitter 500'
@ -10,3 +17,15 @@ stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:cron]
directory=/service
command=sh -c 'python3 -m flask cron'
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[eventlistener:memmon]
command=memmon -p cron=200MB
events=TICK_60

238
files/__main__.py
View file

@ -1,39 +1,59 @@
'''
Main entry point for the application. Global state among other things are
stored here.
'''
from pathlib import Path
import gevent.monkey
gevent.monkey.patch_all()
from os import environ, path
import secrets
from files.helpers.strings import bool_from_string
from flask import *
from flask_caching import Cache
from flask_limiter import Limiter
from flask_compress import Compress
from flask_mail import Mail
# ^ special case: in general imports should go
# stdlib - externals - internals, but gevent does monkey patching for stdlib
# functions so we want to monkey patch before importing other things
import faulthandler
from os import environ
from pathlib import Path
import flask
import flask_caching
import flask_compress
import flask_limiter
import flask_mail
import flask_profiler
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy import *
import gevent
import redis
import time
from sys import stdout, argv
import faulthandler
import json
from sqlalchemy.engine import Engine, create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from files.helpers.config.const import Service
from files.helpers.strings import bool_from_string
app = Flask(__name__, template_folder='templates')
# first, let's parse arguments to find out what type of instance this is...
service:Service = Service.from_argv()
# ...and then let's create our flask app...
app = flask.app.Flask(__name__, template_folder='templates')
app.url_map.strict_slashes = False
app.jinja_env.cache = {}
app.jinja_env.auto_reload = True
faulthandler.enable()
# ...then check that debug mode was not accidentally enabled...
if bool_from_string(environ.get("ENFORCE_PRODUCTION", True)) and app.debug:
raise ValueError("Debug mode is not allowed! If this is a dev environment, please set ENFORCE_PRODUCTION to false")
# ...and then attempt to load a .env file if the environment is not configured...
if environ.get("SITE_ID") is None:
from dotenv import load_dotenv
load_dotenv(dotenv_path=Path("env"))
# ...and let's add the flask profiler if it's enabled...
if environ.get("FLASK_PROFILER_ENDPOINT"):
app.config["flask_profiler"] = {
"enabled": True,
@ -51,12 +71,14 @@ if environ.get("FLASK_PROFILER_ENDPOINT"):
profiler = flask_profiler.Profiler()
profiler.init_app(app)
try:
from easy_profile import EasyProfileMiddleware
from jinja2.utils import internal_code
# ...and then let's install code to unmangle jinja2 stacktraces for easy_profile...
try:
import inspect as inspectlib
import linecache
from easy_profile import EasyProfileMiddleware
from jinja2.utils import internal_code
def jinja_unmangle_stacktrace():
rewritten_frames = []
@ -87,120 +109,110 @@ except ModuleNotFoundError:
# failed to import, just keep on going
pass
app.config["SITE_ID"]=environ.get("SITE_ID").strip()
app.config["SITE_TITLE"]=environ.get("SITE_TITLE").strip()
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['DATABASE_URL'] = environ.get("DATABASE_URL", "postgresql://postgres@localhost:5432")
app.config['SECRET_KEY'] = environ.get('MASTER_KEY')
app.config["SERVER_NAME"] = environ.get("DOMAIN").strip()
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 0 if app.debug else 3153600
app.config["SESSION_COOKIE_NAME"] = "session_" + environ.get("SITE_ID").strip().lower()
app.config["VERSION"] = "1.0.0"
app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024
app.config["SESSION_COOKIE_SECURE"] = bool(environ.get('SESSION_COOKIE_SECURE', "localhost" not in environ.get("DOMAIN")))
app.config["SESSION_COOKIE_SAMESITE"] = "Lax"
app.config["PERMANENT_SESSION_LIFETIME"] = 60 * 60 * 24 * 365
app.config["DEFAULT_COLOR"] = environ.get("DEFAULT_COLOR", "ffffff").strip()
app.config["DEFAULT_THEME"] = "TheMotte"
app.config["FORCE_HTTPS"] = 1
app.config["UserAgent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
app.config["HCAPTCHA_SITEKEY"] = environ.get("HCAPTCHA_SITEKEY","").strip()
app.config["HCAPTCHA_SECRET"] = environ.get("HCAPTCHA_SECRET","").strip()
app.config["SPAM_SIMILARITY_THRESHOLD"] = float(environ.get("SPAM_SIMILARITY_THRESHOLD", 0.5))
app.config["SPAM_URL_SIMILARITY_THRESHOLD"] = float(environ.get("SPAM_URL_SIMILARITY_THRESHOLD", 0.1))
app.config["SPAM_SIMILAR_COUNT_THRESHOLD"] = int(environ.get("SPAM_SIMILAR_COUNT_THRESHOLD", 10))
app.config["COMMENT_SPAM_SIMILAR_THRESHOLD"] = float(environ.get("COMMENT_SPAM_SIMILAR_THRESHOLD", 0.5))
app.config["COMMENT_SPAM_COUNT_THRESHOLD"] = int(environ.get("COMMENT_SPAM_COUNT_THRESHOLD", 10))
app.config["CACHE_TYPE"] = "RedisCache"
app.config["CACHE_REDIS_URL"] = environ.get("REDIS_URL", "redis://localhost")
app.config['MAIL_SERVER'] = environ.get("MAIL_SERVER", "").strip()
app.config['MAIL_PORT'] = 587
app.config['MAIL_USE_TLS'] = True
app.config['MAIL_USERNAME'] = environ.get("MAIL_USERNAME", "").strip()
app.config['MAIL_PASSWORD'] = environ.get("MAIL_PASSWORD", "").strip()
app.config['DESCRIPTION'] = environ.get("DESCRIPTION", "DESCRIPTION GOES HERE").strip()
app.config['SETTINGS'] = {}
app.config['SQLALCHEMY_DATABASE_URI'] = app.config['DATABASE_URL']
app.config['MENTION_LIMIT'] = int(environ.get('MENTION_LIMIT', 100))
app.config['MULTIMEDIA_EMBEDDING_ENABLED'] = environ.get('MULTIMEDIA_EMBEDDING_ENABLED', "false").lower() == "true"
app.config['RESULTS_PER_PAGE_COMMENTS'] = int(environ.get('RESULTS_PER_PAGE_COMMENTS',50))
app.config['SCORE_HIDING_TIME_HOURS'] = int(environ.get('SCORE_HIDING_TIME_HOURS'))
app.config['ENABLE_SERVICES'] = bool_from_string(environ.get('ENABLE_SERVICES', False))
# ...and let's load up app config...
app.config['DBG_VOLUNTEER_PERMISSIVE'] = bool_from_string(environ.get('DBG_VOLUNTEER_PERMISSIVE', False))
app.config['VOLUNTEER_JANITOR_ENABLE'] = bool_from_string(environ.get('VOLUNTEER_JANITOR_ENABLE', True))
from files.helpers.config.const import (DEFAULT_THEME, MAX_CONTENT_LENGTH,
PERMANENT_SESSION_LIFETIME,
SESSION_COOKIE_SAMESITE)
from files.helpers.config.environment import *
r=redis.Redis(host=environ.get("REDIS_URL", "redis://localhost"), decode_responses=True, ssl_cert_reqs=None)
app.config.update({
"SITE_ID": SITE_ID,
"SITE_TITLE": SITE_TITLE,
"SQLALCHEMY_TRACK_MODIFICATIONS": SQLALCHEMY_TRACK_MODIFICATIONS,
"DATABASE_URL": DATABASE_URL,
"SECRET_KEY": SECRET_KEY,
"SERVER_NAME": SERVER_NAME,
"SEND_FILE_MAX_AGE_DEFAULT": 0 if app.debug else 3153600,
"SESSION_COOKIE_NAME": f'session_{SITE_ID}',
"VERSION": "1.0.0",
"MAX_CONTENT_LENGTH": MAX_CONTENT_LENGTH,
"SESSION_COOKIE_SECURE": SESSION_COOKIE_SECURE,
"SESSION_COOKIE_SAMESITE": SESSION_COOKIE_SAMESITE,
"PERMANENT_SESSION_LIFETIME": PERMANENT_SESSION_LIFETIME,
"DEFAULT_COLOR": DEFAULT_COLOR,
"DEFAULT_THEME": DEFAULT_THEME,
"FORCE_HTTPS": 1,
"UserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36",
"HCAPTCHA_SITEKEY": HCAPTCHA_SITEKEY,
"HCAPTCHA_SECRET": HCAPTCHA_SECRET,
"SPAM_SIMILARITY_THRESHOLD": SPAM_SIMILARITY_THRESHOLD,
"SPAM_URL_SIMILARITY_THRESHOLD": SPAM_URL_SIMILARITY_THRESHOLD,
"SPAM_SIMILAR_COUNT_THRESHOLD": SPAM_SIMILAR_COUNT_THRESHOLD,
"COMMENT_SPAM_SIMILAR_THRESHOLD": COMMENT_SPAM_SIMILAR_THRESHOLD,
"COMMENT_SPAM_COUNT_THRESHOLD": COMMENT_SPAM_COUNT_THRESHOLD,
"CACHE_TYPE": "RedisCache",
"CACHE_REDIS_URL": CACHE_REDIS_URL,
"MAIL_SERVER": MAIL_SERVER,
"MAIL_PORT": MAIL_PORT,
"MAIL_USE_TLS": MAIL_USE_TLS,
"DESCRIPTION": DESCRIPTION,
"MAIL_USERNAME": MAIL_USERNAME,
"MAIL_PASSWORD": MAIL_PASSWORD,
"DESCRIPTION": DESCRIPTION,
"SETTINGS": {},
"SQLALCHEMY_DATABASE_URI": DATABASE_URL,
"MENTION_LIMIT": MENTION_LIMIT,
"MULTIMEDIA_EMBEDDING_ENABLED": MULTIMEDIA_EMBEDDING_ENABLED,
"RESULTS_PER_PAGE_COMMENTS": RESULTS_PER_PAGE_COMMENTS,
"SCORE_HIDING_TIME_HOURS": SCORE_HIDING_TIME_HOURS,
"ENABLE_SERVICES": ENABLE_SERVICES,
"RATE_LIMITER_ENABLED": RATE_LIMITER_ENABLED,
"DBG_VOLUNTEER_PERMISSIVE": DBG_VOLUNTEER_PERMISSIVE,
"VOLUNTEER_JANITOR_ENABLE": VOLUNTEER_JANITOR_ENABLE,
})
# ...and then let's load redis so that...
r = redis.Redis(
host=CACHE_REDIS_URL,
decode_responses=True,
ssl_cert_reqs=None
)
# ...we can configure our ratelimiter...
def get_remote_addr():
with app.app_context():
return request.headers.get('X-Real-IP', default='127.0.0.1')
app.config['RATE_LIMITER_ENABLED'] = not bool_from_string(environ.get('DBG_LIMITER_DISABLED', False))
if not app.config['RATE_LIMITER_ENABLED']:
if service.enable_services and not RATE_LIMITER_ENABLED:
print("Rate limiter disabled in debug mode!")
limiter = Limiter(
limiter = flask_limiter.Limiter(
key_func=get_remote_addr,
app=app,
default_limits=["3/second;30/minute;200/hour;1000/day"],
application_limits=["10/second;200/minute;5000/hour;10000/day"],
storage_uri=environ.get("REDIS_URL", "redis://localhost"),
storage_uri=CACHE_REDIS_URL,
auto_check=False,
enabled=app.config['RATE_LIMITER_ENABLED'],
enabled=RATE_LIMITER_ENABLED,
)
engine = create_engine(app.config['DATABASE_URL'])
# ...and then after that we can load the database.
db_session = scoped_session(sessionmaker(bind=engine, autoflush=False, future=True))
engine: Engine = create_engine(DATABASE_URL)
db_session: scoped_session = scoped_session(sessionmaker(
bind=engine,
autoflush=False,
future=True,
))
cache = Cache(app)
Compress(app)
mail = Mail(app)
# now that we've that, let's add the cache, compression, and mail extensions to our app...
@app.before_request
def before_request():
with open('site_settings.json', 'r') as f:
app.config['SETTINGS'] = json.load(f)
cache = flask_caching.Cache(app)
flask_compress.Compress(app)
mail = flask_mail.Mail(app)
if request.host != app.config["SERVER_NAME"]:
return {"error": "Unauthorized host provided."}, 403
# ...and then import the before and after request handlers if this we will import routes.
if not app.config['SETTINGS']['Bots'] and request.headers.get("Authorization"):
abort(403, "Bots are currently not allowed")
if service.enable_services:
from files.routes.allroutes import *
g.agent = request.headers.get("User-Agent")
if not g.agent:
return 'Please use a "User-Agent" header!', 403
# setup is done. let's conditionally import the rest of the routes.
ua = g.agent.lower()
g.debug = app.debug
g.webview = ('; wv) ' in ua)
g.inferior_browser = (
'iphone' in ua or
'ipad' in ua or
'ipod' in ua or
'mac os' in ua or
' firefox/' in ua)
g.timestamp = int(time.time())
limiter.check()
g.db = db_session()
@app.teardown_appcontext
def teardown_request(error):
if hasattr(g, 'db') and g.db:
g.db.close()
stdout.flush()
@app.after_request
def after_request(response):
response.headers.add("Strict-Transport-Security", "max-age=31536000")
response.headers.add("X-Frame-Options", "deny")
return response
if "load_chat" in argv:
from files.routes.chat import *
else:
if service == Service.THEMOTTE:
from files.routes import *
elif service == Service.CHAT:
from files.routes.chat import *

6
files/assets/css/main.css
View file

@ -5171,6 +5171,12 @@ div[id^="reply-edit-"] li > p:first-child {
display: inline;
}
/* settings, etc */
.bordered-section {
border: 1px black solid;
padding: 1em;
}
/***********************
Volunteer Teaser

23
files/classes/__init__.py
View file

@ -35,7 +35,6 @@
# First the import * from places which don't go circular
from sqlalchemy import *
from flask import *
# Then everything except what's in files.*
import pyotp
@ -44,19 +43,14 @@ import re
import time
from copy import deepcopy
from datetime import datetime
from flask import g
from flask import render_template
from flask import g, render_template
from json import loads
from math import floor
from os import environ
from os import environ, remove, path
from os import remove, path
from random import randint
from secrets import token_hex
from sqlalchemy.orm import deferred, aliased
from sqlalchemy.orm import relationship
from sqlalchemy.orm import relationship, deferred
from sqlalchemy.orm import aliased, deferred, relationship
from urllib.parse import urlencode, urlparse, parse_qs
from urllib.parse import urlparse
# It is now safe to define the models
from .alts import Alt
@ -83,14 +77,15 @@ from .usernotes import UserTag, UserNote
from .views import ViewerRelationship
from .votes import Vote, CommentVote
from .volunteer_janitor import VolunteerJanitorRecord
from .cron.tasks import RepeatableTask
from .cron.submission import ScheduledSubmissionTask
from .cron.pycallable import PythonCodeTask
# Then the import * from files.*
from files.helpers.const import *
from files.helpers.config.const import *
from files.helpers.media import *
from files.helpers.lazy import *
from files.helpers.lazy import lazy
from files.helpers.security import *
# Then the specific stuff we don't want stomped on
from files.helpers.lazy import lazy
from files.classes.base import Base
from files.__main__ import app, cache
from files.classes.base import Base, CreatedBase

3
files/classes/alts.py
View file

@ -12,5 +12,4 @@ class Alt(Base):
Index('alts_user2_idx', user2)
def __repr__(self):
return f"<Alt(id={self.id})>"
return f"<{self.__class__.__name__}(id={self.id})>"

3
files/classes/award.py
View file

@ -1,9 +1,8 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from os import environ
from files.helpers.config.const import AWARDS
from files.helpers.lazy import lazy
from files.helpers.const import *
class AwardRelationship(Base):

5
files/classes/badges.py
View file

@ -1,12 +1,9 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from os import environ
from files.helpers.lazy import lazy
from files.helpers.const import *
from files.helpers.config.const import *
from files.helpers.assetcache import assetcache_path
from datetime import datetime
from json import loads
class BadgeDef(Base):
__tablename__ = "badge_defs"

46
files/classes/base.py
View file

@ -1,3 +1,47 @@
from sqlalchemy.orm import declarative_base
import time
from datetime import datetime, timedelta, timezone
from sqlalchemy.orm import declarative_base, declared_attr
from sqlalchemy.schema import Column
from sqlalchemy.sql.sqltypes import Integer
from files.helpers.time import format_age, format_datetime
Base = declarative_base()
class CreatedBase(Base):
__abstract__ = True
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs:
kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
@declared_attr
def created_utc(self):
return Column(Integer, nullable=False)
@property
def created_date(self) -> str:
return self.created_datetime
@property
def created_datetime(self) -> str:
return format_datetime(self.created_utc)
@property
def created_datetime_py(self) -> datetime:
return datetime.fromtimestamp(self.created_utc, tz=timezone.utc)
@property
def age_seconds(self) -> int:
return time.time() - self.created_utc
@property
def age_timedelta(self) -> timedelta:
return datetime.now(tz=timezone.utc) - self.created_datetime_py
@property
def age_string(self) -> str:
return format_age(self.created_utc)

39
files/classes/clients.py
View file

@ -1,12 +1,11 @@
from flask import *
from flask import g
from sqlalchemy import *
from sqlalchemy.orm import relationship
from .submission import Submission
from .comment import Comment
from files.classes.base import Base
from files.helpers.lazy import lazy
from files.helpers.const import *
import time
from files.helpers.config.const import *
class OauthApp(Base):
@ -24,18 +23,8 @@ class OauthApp(Base):
author = relationship("User", viewonly=True)
def __repr__(self): return f"<OauthApp(id={self.id})>"
@property
@lazy
def created_date(self):
return time.strftime("%d %B %Y", time.gmtime(self.created_utc))
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
@property
@lazy
@ -43,30 +32,20 @@ class OauthApp(Base):
@lazy
def idlist(self, page=1):
posts = g.db.query(Submission.id).filter_by(app_id=self.id)
posts=posts.order_by(Submission.created_utc.desc())
posts=posts.offset(100*(page-1)).limit(101)
return [x[0] for x in posts.all()]
@lazy
def comments_idlist(self, page=1):
posts = g.db.query(Comment.id).filter_by(app_id=self.id)
posts=posts.order_by(Comment.created_utc.desc())
posts=posts.offset(100*(page-1)).limit(101)
return [x[0] for x in posts.all()]
class ClientAuth(Base):
__tablename__ = "client_auths"
__table_args__ = (
UniqueConstraint('access_token', name='unique_access'),
@ -78,13 +57,3 @@ class ClientAuth(Base):
user = relationship("User", viewonly=True)
application = relationship("OauthApp", viewonly=True)
@property
@lazy
def created_date(self):
return time.strftime("%d %B %Y", time.gmtime(self.created_utc))
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))

168
files/classes/comment.py
View file

@ -1,32 +1,27 @@
from os import environ
import re
import time
from typing import Literal, Optional
from urllib.parse import urlencode, urlparse, parse_qs
from flask import *
from urllib.parse import parse_qs, urlencode, urlparse
from flask import g
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from files.__main__ import app
from files.classes.votes import CommentVote
from files.helpers.const import *
from files.helpers.content import moderated_body
from files.classes.base import CreatedBase
from files.helpers.config.const import *
from files.helpers.config.environment import SCORE_HIDING_TIME_HOURS, SITE_FULL
from files.helpers.content import (body_displayed,
execute_shadowbanned_fake_votes)
from files.helpers.lazy import lazy
from .flags import CommentFlag
from random import randint
from .votes import CommentVote
from math import floor
from files.helpers.time import format_age
CommentRenderContext = Literal['comments', 'volunteer']
class Comment(Base):
class Comment(CreatedBase):
__tablename__ = "comments"
id = Column(Integer, primary_key=True)
author_id = Column(Integer, ForeignKey("users.id"), nullable=False)
parent_submission = Column(Integer, ForeignKey("submissions.id"))
created_utc = Column(Integer, nullable=False)
edited_utc = Column(Integer, default=0, nullable=False)
is_banned = Column(Boolean, default=False, nullable=False)
ghost = Column(Boolean, default=False, nullable=False)
@ -75,8 +70,6 @@ class Comment(Base):
notes = relationship("UserNote", back_populates="comment")
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs:
kwargs["created_utc"] = int(time.time())
if 'filter_state' not in kwargs:
kwargs['filter_state'] = 'normal'
super().__init__(*args, **kwargs)
@ -86,10 +79,9 @@ class Comment(Base):
@property
@lazy
def should_hide_score(self):
comment_age_seconds = int(time.time()) - self.created_utc
comment_age_hours = comment_age_seconds / (60*60)
return comment_age_hours < app.config['SCORE_HIDING_TIME_HOURS']
def should_hide_score(self) -> bool:
comment_age_hours = self.age_seconds / (60*60)
return comment_age_hours < SCORE_HIDING_TIME_HOURS
def _score_context_str(self, score_type:Literal['score', 'upvotes', 'downvotes'],
context:CommentRenderContext) -> str:
@ -134,79 +126,9 @@ class Comment(Base):
return False
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))
@property
@lazy
def age_string(self):
notif_utc = self.__dict__.get("notif_utc")
if notif_utc:
timestamp = notif_utc
elif self.created_utc:
timestamp = self.created_utc
else:
return None
age = int(time.time()) - timestamp
if age < 60:
return "just now"
elif age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
elif age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
elif age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(timestamp)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if now.tm_mday < ctd.tm_mday:
months -= 1
if months < 12:
return f"{months}mo ago"
else:
years = int(months / 12)
return f"{years}yr ago"
@property
@lazy
def edited_string(self):
age = int(time.time()) - self.edited_utc
if age < 60:
return "just now"
elif age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
elif age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
elif age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(self.edited_utc)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if now.tm_mday < ctd.tm_mday:
months -= 1
if months < 12:
return f"{months}mo ago"
else:
years = int(months / 12)
return f"{years}yr ago"
if not self.edited_utc: return "never"
return format_age(self.edited_utc)
@property
@lazy
@ -361,54 +283,24 @@ class Comment(Base):
return data
def realbody(self, v):
moderated:Optional[str] = moderated_body(self, v)
if moderated: return moderated
if self.post and self.post.club and not (v and (v.paid_dues or v.id in [self.author_id, self.post.author_id])): return f"<p>{CC} ONLY</p>"
body = self.body_html or ""
if body:
if v:
body = body.replace("old.reddit.com", v.reddit)
if v.nitter and not '/i/' in body and '/retweets' not in body: body = body.replace("www.twitter.com", "nitter.net").replace("twitter.com", "nitter.net")
if v and v.controversial:
captured = []
for i in controversial_regex.finditer(body):
if i.group(0) in captured: continue
captured.append(i.group(0))
url = i.group(1)
p = urlparse(url).query
p = parse_qs(p)
if 'sort' not in p: p['sort'] = ['controversial']
url_noquery = url.split('?')[0]
body = body.replace(url, f"{url_noquery}?{urlencode(p, True)}")
if v and v.shadowbanned and v.id == self.author_id and 86400 > time.time() - self.created_utc > 60:
ti = max(int((time.time() - self.created_utc)/60), 1)
maxupvotes = min(ti, 13)
rand = randint(0, maxupvotes)
if self.upvotes < rand:
amount = randint(0, 3)
if amount == 1:
self.upvotes += amount
g.db.add(self)
g.db.commit()
body = body_displayed(self, v, is_html=True)
if v and v.controversial:
captured = []
for i in controversial_regex.finditer(body):
if i.group(0) in captured: continue
captured.append(i.group(0))
url = i.group(1)
p = urlparse(url).query
p = parse_qs(p)
if 'sort' not in p: p['sort'] = ['controversial']
url_noquery = url.split('?')[0]
body = body.replace(url, f"{url_noquery}?{urlencode(p, True)}")
execute_shadowbanned_fake_votes(g.db, self, v) # TODO: put in route handler?
return body
def plainbody(self, v):
moderated:Optional[str] = moderated_body(self, v)
if moderated: return moderated
if self.post and self.post.club and not (v and (v.paid_dues or v.id in [self.author_id, self.post.author_id])): return f"<p>{CC} ONLY</p>"
body = self.body
if not body: return ""
return body
return body_displayed(self, v, is_html=False)
@lazy
def collapse_for_user(self, v, path):

55
files/classes/cron/pycallable.py Normal file
View file

@ -0,0 +1,55 @@
import importlib
from types import ModuleType
from typing import Callable
from sqlalchemy.schema import Column, ForeignKey
from sqlalchemy.sql.sqltypes import Integer, String
from files.classes.cron.tasks import (RepeatableTask, ScheduledTaskType,
TaskRunContext)
__all__ = ('PythonCodeTask',)
class PythonCodeTask(RepeatableTask):
'''
A repeatable task that is naively calls a Python callable. It can access
all of the app state as that is provided to the callee. An example task
that sets the fictional `hams` variable on a `User` to `eggs` is shown
below.
```py
from files.classes.user import User
from files.classes.cron.scheduler import TaskRunContext
from files.helpers.get import get_account
def spam_task(ctx:TaskRunContext):
user:Optional[User] = get_account(1784, graceful=True, db=ctx.db)
if not user: raise Exception("User not found!")
user.hams = "eggs"
ctx.db.commit()
```
The `import_path` and `callable` properties are passed to `importlib`
which then imports the module and then calls the callable with the current
task context.
'''
__tablename__ = "tasks_repeatable_python"
__mapper_args__ = {
"polymorphic_identity": int(ScheduledTaskType.PYTHON_CALLABLE),
}
id = Column(Integer, ForeignKey(RepeatableTask.id), primary_key=True)
import_path = Column(String, nullable=False)
callable = Column(String, nullable=False)
def run_task(self, ctx:TaskRunContext):
self.get_function()(ctx)
def get_function(self) -> Callable[[TaskRunContext], None]:
module:ModuleType = importlib.import_module(self.import_path)
fn = getattr(module, self.callable, None)
if not callable(fn):
raise TypeError("Name either does not exist or is not callable")
return fn

174
files/classes/cron/submission.py Normal file
View file

@ -0,0 +1,174 @@
import functools
from datetime import datetime
from sqlalchemy.orm import relationship
from sqlalchemy.schema import Column, ForeignKey
from sqlalchemy.sql.sqltypes import Boolean, Integer, String, Text
from files.classes.cron.tasks import (RepeatableTask, ScheduledTaskType,
TaskRunContext)
from files.classes.submission import Submission
from files.helpers.config.const import (RENDER_DEPTH_LIMIT,
SUBMISSION_TITLE_LENGTH_MAXIMUM)
from files.helpers.config.environment import SITE_FULL
from files.helpers.content import body_displayed
from files.helpers.lazy import lazy
from files.helpers.sanitize import filter_emojis_only
__all__ = ('ScheduledSubmissionTask',)
class ScheduledSubmissionTask(RepeatableTask):
__tablename__ = "tasks_repeatable_scheduled_submissions"
__mapper_args__ = {
"polymorphic_identity": int(ScheduledTaskType.SCHEDULED_SUBMISSION),
}
id = Column(Integer, ForeignKey(RepeatableTask.id), primary_key=True)
author_id_submission = Column(Integer, ForeignKey("users.id"), nullable=False)
ghost = Column(Boolean, default=False, nullable=False)
private = Column(Boolean, default=False, nullable=False)
over_18 = Column(Boolean, default=False, nullable=False)
is_bot = Column(Boolean, default=False, nullable=False)
title = Column(String(SUBMISSION_TITLE_LENGTH_MAXIMUM), nullable=False)
url = Column(String)
body = Column(Text)
body_html = Column(Text)
flair = Column(String)
embed_url = Column(String)
author = relationship("User", foreign_keys=author_id_submission)
task = relationship(RepeatableTask)
submissions = relationship(Submission,
back_populates="task", order_by="Submission.id.desc()")
def run_task(self, ctx:TaskRunContext) -> None:
submission:Submission = self.make_submission(ctx)
with ctx.app_context():
# TODO: stop using app context (currently required for sanitize and
# username pings)
submission.submit(ctx.db) # TODO: thumbnails
submission.publish()
def make_submission(self, ctx:TaskRunContext) -> Submission:
title:str = self.make_title(ctx.trigger_time)
title_html:str = filter_emojis_only(title, graceful=True)
if len(title_html) > 1500: raise ValueError("Rendered title too large")
return Submission(
created_utc=int(ctx.trigger_time.timestamp()),
private=self.private,
author_id=self.author_id_submission,
over_18=self.over_18,
app_id=None,
is_bot =self.is_bot,
title=title,
title_html=title_html,
url=self.url,
body=self.body,
body_html=self.body_html,
flair=self.flair,
ghost=self.ghost,
filter_state='normal',
embed_url=self.embed_url,
task_id=self.id,
)
def make_title(self, trigger_time:datetime) -> str:
return trigger_time.strftime(self.title)
# properties below here are mocked in order to reuse part of the submission
# HTML template for previewing a submitted task
@property
def deleted_utc(self) -> int:
return int(not self.task.enabled)
@functools.cached_property
def title_html(self) -> str:
'''
This is used as a mock property for display in submission listings that
contain scheduled posts.
.. warning::
This property should not be used for generating the HTML for an actual
submission as this will be missing the special formatting that may be
applies to titles. Instead call
`ScheduledSubmissionContext.make_title()` with the `datetime` that the
event was triggered at.
'''
return filter_emojis_only(self.title)
@property
def author_name(self) -> str:
return self.author.username
@property
def upvotes(self) -> int:
return 1
@property
def score(self) -> int:
return 1
@property
def downvotes(self) -> int:
return 0
@property
def realupvotes(self) -> int:
return 1
@property
def comment_count(self) -> int:
return 0
@property
def views(self) -> int:
return 0
@property
def filter_state(self) -> str:
return 'normal'
def award_count(self, kind):
return 0
@lazy
def realurl(self, v):
return Submission.realurl(self, v)
def realbody(self, v):
return body_displayed(self, v, is_html=True)
def plainbody(self, v):
return body_displayed(self, v, is_html=False)
@lazy
def realtitle(self, v):
return self.title_html if self.title_html else self.title
@lazy
def plaintitle(self, v):
return self.title
@property
def permalink(self):
return f"/tasks/scheduled_posts/{self.id}"
@property
def shortlink(self):
return self.permalink
@property
def is_real_submission(self) -> bool:
return False
@property
def should_hide_score(self) -> bool:
return True
@property
def edit_url(self) -> str:
return f"/tasks/scheduled_posts/{self.id}/content"

398
files/classes/cron/tasks.py Normal file
View file

@ -0,0 +1,398 @@
from __future__ import annotations
import contextlib
import dataclasses
from datetime import date, datetime, timedelta, timezone
from enum import IntEnum, IntFlag
from typing import TYPE_CHECKING, Final, Optional, Union
import flask
import flask_caching
import flask_mail
import redis
from sqlalchemy.orm import relationship, Session
from sqlalchemy.schema import Column, ForeignKey
from sqlalchemy.sql.sqltypes import (Boolean, DateTime, Integer, SmallInteger,
Text, Time)
from files.classes.base import CreatedBase
from files.helpers.time import format_age, format_datetime
if TYPE_CHECKING:
from files.classes.user import User
class ScheduledTaskType(IntEnum):
PYTHON_CALLABLE = 1
SCHEDULED_SUBMISSION = 2
def __str__(self):
if not self.name: return super().__str__()
return self.name.replace('_', ' ').title()
class ScheduledTaskState(IntEnum):
WAITING = 1
'''
A task waiting to be triggered
'''
RUNNING = 2
'''
A task that is currently running
'''
class DayOfWeek(IntFlag):
SUNDAY = 1 << 1
MONDAY = 1 << 2
TUESDAY = 1 << 3
WEDNESDAY = 1 << 4
THURSDAY = 1 << 5
FRIDAY = 1 << 6
SATURDAY = 1 << 7
WEEKDAYS = MONDAY | TUESDAY | WEDNESDAY | THURSDAY | FRIDAY
WEEKENDS = SATURDAY | SUNDAY
NONE = 0 << 0
ALL = WEEKDAYS | WEEKENDS
@classmethod
@property
def all_days(cls) -> list["DayOfWeek"]:
return [
cls.SUNDAY, cls.MONDAY, cls.TUESDAY, cls.WEDNESDAY,
cls.THURSDAY, cls.FRIDAY, cls.SATURDAY
]
@property
def empty(self) -> bool:
return self not in self.ALL
def __contains__(self, other:Union[date, "DayOfWeek"]) -> bool:
_days:dict[int, "DayOfWeek"] = {
0: self.MONDAY,
1: self.TUESDAY,
2: self.WEDNESDAY,
3: self.THURSDAY,
4: self.FRIDAY,
5: self.SATURDAY,
6: self.SUNDAY
}
if not isinstance(other, date):
return super().__contains__(other)
weekday:int = other.weekday()
if not 0 <= weekday <= 6:
raise Exception(
f"Unexpected weekday value (got {weekday}, expected 0-6)")
return _days[weekday] in self
_UserConvertible = Union["User", str, int]
@dataclasses.dataclass(frozen=True, kw_only=True, slots=True)
class TaskRunContext:
'''
A full task run context, with references to all app globals embedded.
This is the entirety of the application's global state at this point.
This is explicit state. This is useful so scheduled tasks do not have
to import from `files.__main__` and so they can use all of the features
of the application without being in a request context.
'''
app:flask.app.Flask
'''
The application. Many of the app functions use the app context globals and
do not have their state explicitly passed. This is a convenience get out of
jail free card so that most features (excepting those that require a
`request` context can be used.)
'''
cache:flask_caching.Cache
'''
A cache extension. This is useful for situations where a scheduled task
might want to interact with the cache in some way (for example invalidating
or adding something to the cache.)
'''
db:Session
'''
A database session. Useful for when a task needs to modify something in the
database (for example creating a submission)
'''
mail:flask_mail.Mail
'''
The mail extension. Needed for sending emails.
'''
redis:redis.Redis
'''
A direct reference to our redis connection. Normally most operations that
involve the redis datastore use flask_caching's Cache object (accessed via
the `cache` property), however this is provided as a convenience for more
granular redis operations.
'''
task:RepeatableTask
'''
A reference to the task that is being ran.
'''
task_run:RepeatableTaskRun
'''
A reference to this current run of the task.
'''
trigger_time:datetime
'''
The date and time (UTC) that this task was triggered
'''
@property
def run_time(self) -> datetime:
'''
The date and time (UTC) that this task was actually ran
'''
return self.task_run.created_datetime_py
@contextlib.contextmanager
def app_context(self, *, v:Optional[_UserConvertible]=None):
'''
Context manager that uses `self.app` to generate an app context and set
up the application with expected globals. This assigns `g.db`, `g.v`,
and `g.debug`.
This is intended for use with legacy code that does not pass state
explicitly and instead relies on the use of `g` for state passing. If
at all possible, state should be passed explicitly to functions that
require it.
Usage is simple:
```py
with ctx.app_context() as app_ctx:
# code that requires g
```
Any code that uses `g` can be ran here. As this is intended for
scenarios that may be outside of a request context code that uses the
request context may still raise `RuntimeException`s.
An example
```py
from flask import g, request # import works ok
def legacy_function():
u:Optional[User] = g.db.get(User, 1784) # works ok! :)
u.admin_level = \\
request.values.get("admin_level", default=9001, type=int)
# raises a RuntimeError :(
g.db.commit()
```
This is because there is no actual request being made. Creating a
mock request context is doable but outside of the scope of this
function as this is often not needed outside of route handlers (where
this function is out of scope anyway).
:param v: A `User`, an `int`, a `str`, or `None`. `g.v` will be set
using the following rules:
1. If `v` is an `int`, `files.helpers.get_account` is called and the
result of that is stored in `g.v`.
2. If `v` is an `str`, `files.helpers.get_user` is called and the
result of that is stored in `g.v`.
3. If `v` is a `User`, it is stored in `g.v`.
It is expected that callees will provide a valid user ID or username.
If an invalid one is provided, *no* exception will be raised and `g.v`
will be set to `None`.
This is mainly provided as an optional feature so that tasks can be
somewhat "sudo"ed as a particular user. Note that `g.v` is always
assigned (even if to `None`) in order to prevent code that depends on
its existence from raising an exception.
'''
with self.app.app_context() as app_ctx:
app_ctx.g.db = self.db
from files.helpers.get import get_account, get_user
if isinstance(v, str):
v = get_user(v, graceful=True)
elif isinstance(v, int):
v = get_account(v, graceful=True, db=self.db)
app_ctx.g.v = v
app_ctx.g.debug = self.app.debug
yield app_ctx
@contextlib.contextmanager
def db_transaction(self):
try:
yield
self.db.commit()
except:
self.db.rollback()
_TABLE_NAME: Final[str] = "tasks_repeatable"
class RepeatableTask(CreatedBase):
__tablename__ = _TABLE_NAME
id = Column(Integer, primary_key=True, nullable=False)
author_id = Column(Integer, ForeignKey("users.id"), nullable=False)
type_id = Column(SmallInteger, nullable=False)
enabled = Column(Boolean, default=True, nullable=False)
run_state = Column(SmallInteger, default=int(ScheduledTaskState.WAITING), nullable=False)
run_time_last = Column(DateTime, default=None)
frequency_day = Column(SmallInteger, nullable=False)
time_of_day_utc = Column(Time, nullable=False)
runs = relationship("RepeatableTaskRun", back_populates="task")
@property
def type(self) -> ScheduledTaskType:
return ScheduledTaskType(self.type_id)
@type.setter
def type(self, value:ScheduledTaskType):
self.type_id = value
@property
def frequency_day_flags(self) -> DayOfWeek:
return DayOfWeek(self.frequency_day)
@frequency_day_flags.setter
def frequency_day_flags(self, value:DayOfWeek):
self.frequency_day = int(value)
@property
def run_state_enum(self) -> ScheduledTaskState:
return ScheduledTaskState(self.run_state)
@run_state_enum.setter
def run_state_enum(self, value:ScheduledTaskState):
self.run_state = int(value)
@property
def run_time_last_or_created_utc(self) -> datetime:
return self.run_time_last or self.created_datetime_py
@property
def run_time_last_str(self) -> str:
if not self.run_time_last: return 'Never'
return (f'{format_datetime(self.run_time_last)} '
f'({format_age(self.run_time_last)})')
@property
def trigger_time(self) -> datetime | None:
return self.next_trigger(self.run_time_last_or_created_utc)
def can_run(self, now: datetime) -> bool:
return not (
self.trigger_time is None
or now < self.trigger_time
or self.run_state_enum != ScheduledTaskState.WAITING)
def next_trigger(self, anchor: datetime) -> datetime | None:
if not self.enabled: return None
if self.frequency_day_flags.empty: return None
day:timedelta = timedelta(1.0)
target_date:datetime = anchor - day # incremented at start of for loop
for i in range(8):
target_date = target_date + day
if i == 0 and target_date.time() > self.time_of_day_utc: continue
if target_date in self.frequency_day_flags: break
else:
raise Exception("Could not find suitable timestamp to run next task")
return datetime.combine(target_date, self.time_of_day_utc, tzinfo=timezone.utc) # type: ignore
def run(self, db: Session, trigger_time: datetime) -> RepeatableTaskRun:
run:RepeatableTaskRun = RepeatableTaskRun(task_id=self.id)
try:
from files.__main__ import app, cache, mail, r # i know
ctx: TaskRunContext = TaskRunContext(
app=app,
cache=cache,
db=db,
mail=mail,
redis=r,
task=self,
task_run=run,
trigger_time=trigger_time,
)
self.run_task(ctx)
except Exception as e:
run.exception = e
run.completed_utc = datetime.now(tz=timezone.utc)
db.add(run)
return run
def run_task(self, ctx:TaskRunContext):
raise NotImplementedError()
def contains_day_str(self, day_str:str) -> bool:
return (bool(day_str)
and DayOfWeek[day_str.upper()] in self.frequency_day_flags)
def __repr__(self) -> str:
return f'<{self.__class__.__name__}(id={self.id}, created_utc={self.created_date}, author_id={self.author_id})>'
__mapper_args__ = {
"polymorphic_on": type_id,
}
class RepeatableTaskRun(CreatedBase):
__tablename__ = "tasks_repeatable_runs"
id = Column(Integer, primary_key=True)
task_id = Column(Integer, ForeignKey(RepeatableTask.id), nullable=False)
manual = Column(Boolean, default=False, nullable=False)
traceback_str = Column(Text, nullable=True)
completed_utc = Column(DateTime)
task = relationship(RepeatableTask, back_populates="runs")
_exception: Optional[Exception] = None # not part of the db model
@property
def completed_datetime_py(self) -> datetime | None:
if self.completed_utc is None:
return None
return datetime.combine(
self.completed_utc.date(),
self.completed_utc.time(),
timezone.utc)
@property
def completed_datetime_str(self) -> str:
return format_datetime(self.completed_utc)
@property
def status_text(self) -> str:
if not self.completed_utc: return "Running"
return "Failed" if self.traceback_str else "Completed"
@property
def time_elapsed(self) -> Optional[timedelta]:
if self.completed_datetime_py is None: return None
return self.completed_datetime_py - self.created_datetime_py
@property
def time_elapsed_str(self) -> str:
elapsed:Optional[timedelta] = self.time_elapsed
if not elapsed: return ''
return str(elapsed)
@property
def exception(self) -> Optional[Exception]:
return self._exception
@exception.setter
def exception(self, value: Optional[Exception]) -> None:
self._exception = value
self.traceback_str = str(value) if value else None

1
files/classes/domains.py
View file

@ -2,7 +2,6 @@ from sqlalchemy import *
from files.classes.base import Base
class BannedDomain(Base):
__tablename__ = "banneddomains"
domain = Column(String, primary_key=True)
reason = Column(String, nullable=False)

41
files/classes/flags.py
View file

@ -1,77 +1,44 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from files.classes.base import CreatedBase
from files.helpers.lazy import lazy
from files.helpers.const import *
import time
class Flag(Base):
from files.helpers.config.const import *
class Flag(CreatedBase):
__tablename__ = "flags"
id = Column(Integer, primary_key=True)
post_id = Column(Integer, ForeignKey("submissions.id"), nullable=False)
user_id = Column(Integer, ForeignKey("users.id"), nullable=False)
reason = Column(String)
created_utc = Column(Integer, nullable=False)
Index('flag_user_idx', user_id)
user = relationship("User", primaryjoin = "Flag.user_id == User.id", uselist = False, viewonly=True)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
@property
@lazy
def created_date(self):
return time.strftime("%d %B %Y", time.gmtime(self.created_utc))
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))
@lazy
def realreason(self, v):
return self.reason
class CommentFlag(Base):
class CommentFlag(CreatedBase):
__tablename__ = "commentflags"
id = Column(Integer, primary_key=True)
comment_id = Column(Integer, ForeignKey("comments.id"), nullable=False)
user_id = Column(Integer, ForeignKey("users.id"), nullable=False)
reason = Column(String)
created_utc = Column(Integer, nullable=False)
Index('cflag_user_idx', user_id)
user = relationship("User", primaryjoin = "CommentFlag.user_id == User.id", uselist = False, viewonly=True)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
@property
@lazy
def created_date(self):
return time.strftime("%d %B %Y", time.gmtime(self.created_utc))
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))
@lazy
def realreason(self, v):
return self.reason

15
files/classes/follows.py
View file

@ -1,22 +1,19 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
import time
from files.classes.base import CreatedBase
class Follow(Base):
class Follow(CreatedBase):
__tablename__ = "follows"
target_id = Column(Integer, ForeignKey("users.id"), primary_key=True)
user_id = Column(Integer, ForeignKey("users.id"), primary_key=True)
created_utc = Column(Integer, nullable=False)
Index('follow_user_id_index', user_id)
user = relationship("User", uselist=False, primaryjoin="User.id==Follow.user_id", viewonly=True)
target = relationship("User", primaryjoin="User.id==Follow.target_id", viewonly=True)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
return (
f"<{self.__class__.__name__}("
f"target_id={self.target_id}, user_id={self.user_id})>"
)

22
files/classes/leaderboard.py
View file

@ -2,9 +2,9 @@ from dataclasses import dataclass
from typing import Any, Callable, Final, Optional
from sqlalchemy import Column, func
from sqlalchemy.orm import scoped_session, Query
from sqlalchemy.orm import Session, Query
from files.helpers.const import LEADERBOARD_LIMIT
from files.helpers.config.const import LEADERBOARD_LIMIT
from files.classes.badges import Badge
from files.classes.marsey import Marsey
@ -51,9 +51,9 @@ class Leaderboard:
raise NotImplementedError()
class SimpleLeaderboard(Leaderboard):
def __init__(self, v:User, meta:LeaderboardMeta, db:scoped_session, users_query:Query, column:Column):
def __init__(self, v:User, meta:LeaderboardMeta, db:Session, users_query:Query, column:Column):
super().__init__(v, meta)
self.db:scoped_session = db
self.db:Session = db
self.users_query:Query = users_query
self.column:Column = column
self._calculate()
@ -92,9 +92,9 @@ class _CountedAndRankedLeaderboard(Leaderboard):
return func.rank().over(order_by=func.count(criteria).desc()).label("rank")
class BadgeMarseyLeaderboard(_CountedAndRankedLeaderboard):
def __init__(self, v:User, meta:LeaderboardMeta, db:scoped_session, column:Column):
def __init__(self, v:User, meta:LeaderboardMeta, db:Session, column:Column):
super().__init__(v, meta)
self.db:scoped_session = db
self.db:Session = db
self.column = column
self._calculate()
@ -135,9 +135,9 @@ class BadgeMarseyLeaderboard(_CountedAndRankedLeaderboard):
return lambda u:self._all_users[u]
class UserBlockLeaderboard(_CountedAndRankedLeaderboard):
def __init__(self, v:User, meta:LeaderboardMeta, db:scoped_session, column:Column):
def __init__(self, v:User, meta:LeaderboardMeta, db:Session, column:Column):
super().__init__(v, meta)
self.db:scoped_session = db
self.db:Session = db
self.column = column
self._calculate()
@ -169,7 +169,7 @@ class UserBlockLeaderboard(_CountedAndRankedLeaderboard):
return self._v_value
class RawSqlLeaderboard(Leaderboard):
def __init__(self, meta:LeaderboardMeta, db:scoped_session, query:str) -> None: # should be LiteralString on py3.11+
def __init__(self, meta:LeaderboardMeta, db:Session, query:str) -> None: # should be LiteralString on py3.11+
super().__init__(None, meta)
self.db = db
self._calculate(query)
@ -234,7 +234,7 @@ FROM cv_for_user cvfu
ORDER BY count DESC LIMIT 25
"""
def __init__(self, meta:LeaderboardMeta, db:scoped_session) -> None:
def __init__(self, meta:LeaderboardMeta, db:Session) -> None:
super().__init__(meta, db, self._query)
class GivenUpvotesLeaderboard(RawSqlLeaderboard):
@ -248,5 +248,5 @@ FULL OUTER JOIN (SELECT user_id, COUNT(*) FROM commentvotes WHERE vote_type = 1
ORDER BY count DESC LIMIT 25
"""
def __init__(self, meta:LeaderboardMeta, db:scoped_session) -> None:
def __init__(self, meta:LeaderboardMeta, db:Session) -> None:
super().__init__(meta, db, self._query)

16
files/classes/mod.py
View file

@ -1,26 +1,14 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from files.classes.base import CreatedBase
from files.helpers.lazy import *
import time
class Mod(Base):
class Mod(CreatedBase):
__tablename__ = "mods"
user_id = Column(Integer, ForeignKey("users.id"), primary_key=True)
sub = Column(String, ForeignKey("subs.name"), primary_key=True)
created_utc = Column(Integer, nullable=False)
Index('fki_mod_sub_fkey', sub)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(user_id={self.user_id}, sub={self.sub})>"
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))

63
files/classes/mod_logs.py
View file

@ -1,14 +1,15 @@
import logging
from copy import deepcopy
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
import time
from files.helpers.lazy import lazy
from os import environ
from copy import deepcopy
from files.helpers.const import *
import logging
class ModAction(Base):
from files.classes.base import CreatedBase
from files.helpers.config.const import *
from files.helpers.lazy import lazy
class ModAction(CreatedBase):
__tablename__ = "modactions"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("users.id"))
@ -17,7 +18,6 @@ class ModAction(Base):
target_submission_id = Column(Integer, ForeignKey("submissions.id"))
target_comment_id = Column(Integer, ForeignKey("comments.id"))
_note=Column(String)
created_utc = Column(Integer, nullable=False)
Index('fki_modactions_user_fkey', target_user_id)
Index('modaction_action_idx', kind)
@ -29,49 +29,12 @@ class ModAction(Base):
target_user = relationship("User", primaryjoin="User.id==ModAction.target_user_id", viewonly=True)
target_post = relationship("Submission", viewonly=True)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
@property
@lazy
def age_string(self):
age = int(time.time()) - self.created_utc
if age < 60:
return "just now"
elif age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
elif age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
elif age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(self.created_utc)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if now.tm_mday < ctd.tm_mday:
months -= 1
if months < 12:
return f"{months}mo ago"
else:
years = int(months / 12)
return f"{years}yr ago"
@property
def note(self):
if self.kind=="ban_user":
if self.kind == "ban_user":
if self.target_post: return f'for <a href="{self.target_post.permalink}">post</a>'
elif self.target_comment_id: return f'for <a href="/comment/{self.target_comment_id}">comment</a>'
else: return self._note
@ -92,11 +55,9 @@ class ModAction(Base):
@property
@lazy
def string(self):
output = self.lookup_action_type()["str"].format(self=self, cc=CC_TITLE)
if self.note: output += f" <i>({self.note})</i>"
if not self.note: return output
output += f" <i>({self.note})</i>"
return output
@property

13
files/classes/notifications.py
View file

@ -1,16 +1,13 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
import time
class Notification(Base):
from files.classes.base import CreatedBase
class Notification(CreatedBase):
__tablename__ = "notifications"
user_id = Column(Integer, ForeignKey("users.id"), primary_key=True)
comment_id = Column(Integer, ForeignKey("comments.id"), primary_key=True)
read = Column(Boolean, default=False, nullable=False)
created_utc = Column(Integer, nullable=False)
Index('notification_read_idx', read)
Index('notifications_comment_idx', comment_id)
@ -19,9 +16,5 @@ class Notification(Base):
comment = relationship("Comment", viewonly=True)
user = relationship("User", viewonly=True)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
return f"<{self.__class__.__name__}(user_id={self.user_id}, comment_id={self.comment_id})>"

15
files/classes/saves.py
View file

@ -4,21 +4,18 @@ from files.classes.base import Base
class SaveRelationship(Base):
__tablename__ = "save_relationship"
__tablename__="save_relationship"
user_id=Column(Integer, ForeignKey("users.id"), primary_key=True)
submission_id=Column(Integer, ForeignKey("submissions.id"), primary_key=True)
user_id = Column(Integer, ForeignKey("users.id"), primary_key=True)
submission_id = Column(Integer, ForeignKey("submissions.id"), primary_key=True)
Index('fki_save_relationship_submission_fkey', submission_id)
class CommentSaveRelationship(Base):
__tablename__ = "comment_save_relationship"
__tablename__="comment_save_relationship"
user_id=Column(Integer, ForeignKey("users.id"), primary_key=True)
comment_id=Column(Integer, ForeignKey("comments.id"), primary_key=True)
user_id = Column(Integer, ForeignKey("users.id"), primary_key=True)
comment_id = Column(Integer, ForeignKey("comments.id"), primary_key=True)
Index('fki_comment_save_relationship_comment_fkey', comment_id)

7
files/classes/sub.py
View file

@ -1,16 +1,12 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.helpers.config.environment import SITE_FULL
from files.classes.base import Base
from files.helpers.lazy import lazy
from os import environ
from .sub_block import *
SITE = environ.get("DOMAIN", '').strip()
if SITE == "localhost": SITE_FULL = 'http://' + SITE
else: SITE_FULL = 'https://' + SITE
class Sub(Base):
__tablename__ = "subs"
name = Column(String, primary_key=True)
sidebar = Column(String)
@ -23,7 +19,6 @@ class Sub(Base):
blocks = relationship("SubBlock", lazy="dynamic", primaryjoin="SubBlock.sub==Sub.name", viewonly=True)
def __repr__(self):
return f"<{self.__class__.__name__}(name={self.name})>"

232
files/classes/submission.py
View file

@ -1,31 +1,30 @@
from os import environ
import random
import re
import time
from typing import Optional
from urllib.parse import urlparse
from flask import render_template
from sqlalchemy import *
from sqlalchemy.orm import relationship, deferred
from files.classes.base import Base
from files.__main__ import app
from files.helpers.const import *
from files.helpers.content import moderated_body
from files.helpers.lazy import lazy
from files.helpers.assetcache import assetcache_path
from .flags import Flag
from .comment import Comment
from flask import g
from .sub import Sub
from .votes import CommentVote
class Submission(Base):
from flask import g
from sqlalchemy import *
from sqlalchemy.orm import (declared_attr, deferred, relationship,
Session)
from files.classes.base import CreatedBase
from files.classes.votes import Vote
from files.helpers.assetcache import assetcache_path
from files.helpers.config.const import *
from files.helpers.config.environment import (SCORE_HIDING_TIME_HOURS, SITE,
SITE_FULL, SITE_ID)
from files.helpers.content import body_displayed
from files.helpers.lazy import lazy
from files.helpers.time import format_age, format_datetime
from .flags import Flag
class Submission(CreatedBase):
__tablename__ = "submissions"
id = Column(Integer, primary_key=True)
author_id = Column(Integer, ForeignKey("users.id"), nullable=False)
edited_utc = Column(Integer, default=0, nullable=False)
created_utc = Column(Integer, nullable=False)
thumburl = Column(String)
is_banned = Column(Boolean, default=False, nullable=False)
bannedfor = Column(Boolean)
@ -56,17 +55,29 @@ class Submission(Base):
ban_reason = Column(String)
embed_url = Column(String)
filter_state = Column(String, nullable=False)
task_id = Column(Integer, ForeignKey("tasks_repeatable_scheduled_submissions.id"))
Index('fki_submissions_approver_fkey', is_approved)
Index('post_app_id_idx', app_id)
Index('subimssion_binary_group_idx', is_banned, deleted_utc, over_18)
Index('submission_isbanned_idx', is_banned)
Index('submission_isdeleted_idx', deleted_utc)
Index('submission_new_sort_idx', is_banned, deleted_utc, created_utc.desc(), over_18)
@declared_attr
def submission_new_sort_idx(self):
return Index('submission_new_sort_idx', self.is_banned, self.deleted_utc, self.created_utc.desc(), self.over_18)
Index('submission_pinned_idx', is_pinned)
Index('submissions_author_index', author_id)
Index('submissions_created_utc_asc_idx', created_utc.nullsfirst())
Index('submissions_created_utc_desc_idx', created_utc.desc())
@declared_attr
def submisission_created_utc_asc_idx(self):
return Index('submissions_created_utc_asc_idx', self.created_utc.nulls_first())
@declared_attr
def submissions_created_utc_desc_idx(self):
return Index('submissions_created_utc_desc_idx', self.created_utc.desc())
Index('submissions_over18_index', over_18)
author = relationship("User", primaryjoin="Submission.author_id==User.id")
@ -77,12 +88,46 @@ class Submission(Base):
comments = relationship("Comment", primaryjoin="Comment.parent_submission==Submission.id")
subr = relationship("Sub", primaryjoin="foreign(Submission.sub)==remote(Sub.name)", viewonly=True)
notes = relationship("UserNote", back_populates="post")
task = relationship("ScheduledSubmissionTask", back_populates="submissions")
bump_utc = deferred(Column(Integer, server_default=FetchedValue()))
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def submit(self, db: Session):
# create submission...
db.add(self)
db.flush()
# then create vote...
vote = Vote(
user_id=self.author_id,
vote_type=1,
submission_id=self.id
)
db.add(vote)
author = self.author
author.post_count = db.query(Submission.id).filter_by(
author_id=self.author_id,
is_banned=False,
deleted_utc=0).count()
db.add(author)
def publish(self):
# this is absolutely horrifying. imports are very very tangled and the
# spider web of imports is hard to maintain. we defer loading these
# imports until as late as possible. otherwise there are import loops
# that would require much more work to untangle
from files.helpers.alerts import notify_submission_publish
from files.helpers.caching import invalidate_cache
if self.private: return
if not self.ghost:
notify_submission_publish(self)
invalidate_cache(
frontlist=True,
userpagelisting=True,
changeloglist=("[changelog]" in self.title.lower()
or "(changelog)" in self.title.lower()),
)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
@ -90,9 +135,8 @@ class Submission(Base):
@property
@lazy
def should_hide_score(self):
submission_age_seconds = int(time.time()) - self.created_utc
submission_age_hours = submission_age_seconds / (60*60)
return submission_age_hours < app.config['SCORE_HIDING_TIME_HOURS']
submission_age_hours = self.age_seconds / (60*60)
return submission_age_hours < SCORE_HIDING_TIME_HOURS
@property
@lazy
@ -110,82 +154,12 @@ class Submission(Base):
return flags
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))
@property
@lazy
def age_string(self):
age = int(time.time()) - self.created_utc
if age < 60:
return "just now"
elif age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
elif age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
elif age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(self.created_utc)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if now.tm_mday < ctd.tm_mday:
months -= 1
if months < 12:
return f"{months}mo ago"
else:
years = int(months / 12)
return f"{years}yr ago"
@property
@lazy
def edited_string(self):
if not self.edited_utc: return "never"
age = int(time.time()) - self.edited_utc
if age < 60:
return "just now"
elif age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
elif age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
elif age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(self.edited_utc)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if months < 12:
return f"{months}mo ago"
else:
years = now.tm_year - ctd.tm_year
return f"{years}yr ago"
return format_age(self.edited_utc) if self.edited_utc else "never"
@property
@lazy
def edited_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.edited_utc))
return format_datetime(self.edited_utc) if self.edited_utc else ""
@property
@lazy
@ -197,7 +171,6 @@ class Submission(Base):
def fullname(self):
return f"t2_{self.id}"
@property
@lazy
def shortlink(self):
@ -298,7 +271,6 @@ class Submission(Base):
@property
@lazy
def json_core(self):
if self.is_banned:
return {'is_banned': True,
'deleted_utc': self.deleted_utc,
@ -320,7 +292,6 @@ class Submission(Base):
@property
@lazy
def json(self):
data=self.json_core
if self.deleted_utc or self.is_banned:
@ -360,50 +331,14 @@ class Submission(Base):
else: return ""
def realbody(self, v):
moderated:Optional[str] = moderated_body(self, v)
if moderated: return moderated
if self.club and not (v and (v.paid_dues or v.id == self.author_id)): return f"<p>{CC} ONLY</p>"
body = self.body_html or ""
if v:
body = body.replace("old.reddit.com", v.reddit)
if v.nitter and '/i/' not in body and '/retweets' not in body: body = body.replace("www.twitter.com", "nitter.net").replace("twitter.com", "nitter.net")
if v and v.shadowbanned and v.id == self.author_id and 86400 > time.time() - self.created_utc > 20:
ti = max(int((time.time() - self.created_utc)/60), 1)
maxupvotes = min(ti, 11)
rand = random.randint(0, maxupvotes)
if self.upvotes < rand:
amount = random.randint(0, 3)
if amount == 1:
self.views += amount*random.randint(3, 5)
self.upvotes += amount
g.db.add(self)
g.db.commit()
return body
return body_displayed(self, v, is_html=True)
def plainbody(self, v):
moderated:Optional[str] = moderated_body(self, v)
if moderated: return moderated
if self.club and not (v and (v.paid_dues or v.id == self.author_id)): return f"<p>{CC} ONLY</p>"
body = self.body
if not body: return ""
if v:
body = body.replace("old.reddit.com", v.reddit)
if v.nitter and '/i/' not in body and '/retweets' not in body: body = body.replace("www.twitter.com", "nitter.net").replace("twitter.com", "nitter.net")
return body
return body_displayed(self, v, is_html=False)
@lazy
def realtitle(self, v):
if self.title_html:
return self.title_html
else:
return self.title
return self.title_html if self.title_html else self.title
@lazy
def plaintitle(self, v):
@ -422,4 +357,13 @@ class Submission(Base):
return False
@lazy
def active_flags(self, v): return len(self.flags(v))
def active_flags(self, v):
return len(self.flags(v))
@property
def is_real_submission(self) -> bool:
return True
@property
def edit_url(self) -> str:
return f"/edit_post/{self.id}"

152
files/classes/user.py
View file

@ -1,36 +1,41 @@
from sqlalchemy.orm import deferred, aliased
from secrets import token_hex
import pyotp
from files.classes.base import Base
from files.helpers.media import *
from files.helpers.const import *
from .alts import Alt
from .saves import *
from .notifications import Notification
from .award import AwardRelationship
from .follows import *
from .subscriptions import *
from .userblock import *
from .badges import *
from .clients import *
from .mod_logs import *
from .mod import *
from .exiles import *
from .sub_block import *
from files.__main__ import app, cache
from files.helpers.security import *
from files.helpers.assetcache import assetcache_path
from files.helpers.contentsorting import apply_time_filter, sort_objects
import random
from datetime import datetime
from os import environ, remove, path
from __future__ import annotations
import time
from typing import TYPE_CHECKING, Union
import pyotp
from flask import g, session
from sqlalchemy.orm import aliased, declared_attr, deferred, relationship
from files.classes.alts import Alt
from files.classes.award import AwardRelationship
from files.classes.badges import Badge
from files.classes.base import CreatedBase
from files.classes.clients import * # note: imports Comment and Submission
from files.classes.exiles import *
from files.classes.follows import Follow
from files.classes.mod import Mod
from files.classes.mod_logs import ModAction
from files.classes.notifications import Notification
from files.classes.saves import CommentSaveRelationship, SaveRelationship
from files.classes.sub_block import SubBlock
from files.classes.subscriptions import Subscription
from files.classes.userblock import UserBlock
from files.helpers.assetcache import assetcache_path
from files.helpers.config.const import *
from files.helpers.config.environment import (CARD_VIEW,
CLUB_TRUESCORE_MINIMUM,
DEFAULT_COLOR,
DEFAULT_TIME_FILTER, SITE_FULL,
SITE_ID)
from files.helpers.security import *
if TYPE_CHECKING:
from files.classes.cron.submission import ScheduledSubmissionTask
defaulttheme = "TheMotte"
defaulttimefilter = environ.get("DEFAULT_TIME_FILTER", "all").strip()
cardview = bool(int(environ.get("CARD_VIEW", 1)))
class User(Base):
class User(CreatedBase):
__tablename__ = "users"
__table_args__ = (
UniqueConstraint('bannerurl', name='one_banner'),
@ -46,7 +51,7 @@ class User(Base):
titlecolor = Column(String(length=6), default=DEFAULT_COLOR, nullable=False)
theme = Column(String, default=defaulttheme, nullable=False)
themecolor = Column(String, default=DEFAULT_COLOR, nullable=False)
cardview = Column(Boolean, default=cardview, nullable=False)
cardview = Column(Boolean, default=CARD_VIEW, nullable=False)
highres = Column(String)
profileurl = Column(String)
bannerurl = Column(String)
@ -63,7 +68,6 @@ class User(Base):
post_count = Column(Integer, default=0, nullable=False)
comment_count = Column(Integer, default=0, nullable=False)
received_award_count = Column(Integer, default=0, nullable=False)
created_utc = Column(Integer, nullable=False)
admin_level = Column(Integer, default=0, nullable=False)
coins_spent = Column(Integer, default=0, nullable=False)
lootboxes_bought = Column(Integer, default=0, nullable=False)
@ -103,7 +107,7 @@ class User(Base):
stored_subscriber_count = Column(Integer, default=0, nullable=False)
defaultsortingcomments = Column(String, default="new", nullable=False)
defaultsorting = Column(String, default="new", nullable=False)
defaulttime = Column(String, default=defaulttimefilter, nullable=False)
defaulttime = Column(String, default=DEFAULT_TIME_FILTER, nullable=False)
is_nofollow = Column(Boolean, default=False, nullable=False)
custom_filter_list = Column(String)
ban_evade = Column(Integer, default=0, nullable=False)
@ -128,7 +132,11 @@ class User(Base):
Index('fki_user_referrer_fkey', referred_by)
Index('user_banned_idx', is_banned)
Index('user_private_idx', is_private)
Index('users_created_utc_index', created_utc)
@declared_attr
def users_created_utc_index(self):
return Index('users_created_utc_index', self.created_utc)
Index('users_subs_idx', stored_subscriber_count)
Index('users_unbanutc_idx', unban_utc.desc())
@ -145,27 +153,24 @@ class User(Base):
notes = relationship("UserNote", foreign_keys='UserNote.reference_user', back_populates="user")
def __init__(self, **kwargs):
if "password" in kwargs:
kwargs["passhash"] = self.hash_password(kwargs["password"])
kwargs.pop("password")
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(**kwargs)
def can_manage_reports(self):
return self.admin_level > 1
def should_comments_be_filtered(self):
from files.__main__ import app # avoiding import loop
if self.admin_level > 0:
return False
# TODO: move settings out of app.config
site_settings = app.config['SETTINGS']
minComments = site_settings.get('FilterCommentsMinComments', 0)
minKarma = site_settings.get('FilterCommentsMinKarma', 0)
minAge = site_settings.get('FilterCommentsMinAgeDays', 0)
accountAgeDays = (datetime.now() - datetime.fromtimestamp(self.created_utc)).days
accountAgeDays = self.age_timedelta.days
return self.comment_count < minComments or accountAgeDays < minAge or self.truecoins < minKarma
@lazy
@ -197,29 +202,6 @@ class User(Base):
def csslazy(self):
return self.css
@property
@lazy
def created_date(self):
return time.strftime("%d %b %Y", time.gmtime(self.created_utc))
@property
@lazy
def discount(self):
if self.patron == 1: discount = 0.90
elif self.patron == 2: discount = 0.85
elif self.patron == 3: discount = 0.80
elif self.patron == 4: discount = 0.75
elif self.patron == 5: discount = 0.70
elif self.patron == 6: discount = 0.65
else: discount = 1
for badge in [69,70,71,72,73]:
if self.has_badge(badge): discount -= discounts[badge]
return discount
@property
@lazy
def user_awards(self):
@ -239,7 +221,7 @@ class User(Base):
@property
@lazy
def paid_dues(self):
return not self.shadowbanned and not (self.is_banned and not self.unban_utc) and (self.admin_level or self.club_allowed or (self.club_allowed != False and self.truecoins > dues))
return not self.shadowbanned and not (self.is_banned and not self.unban_utc) and (self.admin_level or self.club_allowed or (self.club_allowed != False and self.truecoins > CLUB_TRUESCORE_MINIMUM))
@lazy
def any_block_exists(self, other):
@ -253,11 +235,6 @@ class User(Base):
x = pyotp.TOTP(self.mfa_secret)
return x.verify(token, valid_window=1)
@property
@lazy
def age(self):
return int(time.time()) - self.created_utc
@property
@lazy
def ban_reason_link(self):
@ -280,23 +257,6 @@ class User(Base):
if u.patron: return True
return False
@cache.memoize(timeout=86400)
def userpagelisting(self, site=None, v=None, page=1, sort="new", t="all"):
if self.shadowbanned and not (v and (v.admin_level > 1 or v.id == self.id)): return []
posts = g.db.query(Submission.id).filter_by(author_id=self.id, is_pinned=False)
if not (v and (v.admin_level > 1 or v.id == self.id)):
posts = posts.filter_by(deleted_utc=0, is_banned=False, private=False, ghost=False)
posts = apply_time_filter(posts, t, Submission)
posts = sort_objects(posts, sort, Submission)
posts = posts.offset(25 * (page - 1)).limit(26).all()
return [x[0] for x in posts]
@property
@lazy
def follow_count(self):
@ -332,7 +292,6 @@ class User(Base):
@property
@lazy
def formkey(self):
msg = f"{session['session_id']}+{self.id}+{self.login_nonce}"
return generate_hash(msg)
@ -440,7 +399,6 @@ class User(Base):
@property
@lazy
def alts(self):
subq = g.db.query(Alt).filter(
or_(
Alt.user1 == self.id,
@ -477,7 +435,6 @@ class User(Base):
return modded_subs
def has_follower(self, user):
return g.db.query(Follow).filter_by(target_id=self.id, user_id=user.id).one_or_none()
@property
@ -542,8 +499,6 @@ class User(Base):
@property
@lazy
def json_core(self):
now = int(time.time())
if self.is_suspended:
return {'username': self.username,
'url': self.url,
@ -575,8 +530,6 @@ class User(Base):
self.is_banned = admin.id if admin else AUTOJANNY_ID
if reason: self.ban_reason = reason
@property
def is_suspended(self):
return (self.is_banned and (self.unban_utc == 0 or self.unban_utc > time.time()))
@ -590,11 +543,6 @@ class User(Base):
def applications(self):
return g.db.query(OauthApp).filter_by(author_id=self.id).order_by(OauthApp.id)
@property
@lazy
def created_datetime(self):
return time.strftime("%d/%B/%Y %H:%M:%S UTC", time.gmtime(self.created_utc))
@lazy
def subscribed_idlist(self, page=1):
posts = g.db.query(Subscription.submission_id).filter_by(user_id=self.id).all()
@ -614,7 +562,6 @@ class User(Base):
@lazy
def saved_idlist(self, page=1):
saved = [x[0] for x in g.db.query(SaveRelationship.submission_id).filter_by(user_id=self.id).all()]
posts = g.db.query(Submission.id).filter(Submission.id.in_(saved), Submission.is_banned == False, Submission.deleted_utc == 0)
@ -625,7 +572,6 @@ class User(Base):
@lazy
def saved_comment_idlist(self, page=1):
saved = [x[0] for x in g.db.query(CommentSaveRelationship.comment_id).filter_by(user_id=self.id).all()]
comments = g.db.query(Comment.id).filter(Comment.id.in_(saved), Comment.is_banned == False, Comment.deleted_utc == 0)
@ -653,6 +599,14 @@ class User(Base):
# Permissions
def can_edit(self, target:Union[Submission, ScheduledSubmissionTask]):
if isinstance(target, Submission):
if self.id == target.author_id: return True
return self.admin_level >= PERMS['POST_EDITING']
if target.__class__.__name__ == "ScheduledSubmissionTask":
# XXX: avoiding import loop. should be fixed someday.
return self.admin_level >= PERMS['SCHEDULER_POSTS']
@property
def can_see_shadowbanned(self):
return self.admin_level >= PERMS['USER_SHADOWBAN'] or self.shadowbanned

15
files/classes/usernotes.py
View file

@ -1,9 +1,7 @@
import time
from flask import *
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from files.helpers.const import *
from files.classes.base import CreatedBase
from files.helpers.config.const import *
from enum import Enum
from sqlalchemy import Enum as EnumType
@ -17,13 +15,11 @@ class UserTag(Enum):
Spam = 6
Bot = 7
class UserNote(Base):
class UserNote(CreatedBase):
__tablename__ = "usernotes"
id = Column(Integer, primary_key=True)
author_id = Column(Integer, ForeignKey("users.id"), nullable=False)
created_utc = Column(Integer, nullable=False)
reference_user = Column(Integer, ForeignKey("users.id", ondelete='CASCADE'), nullable=False)
reference_comment = Column(Integer, ForeignKey("comments.id", ondelete='SET NULL'))
reference_post = Column(Integer, ForeignKey("submissions.id", ondelete='SET NULL'))
@ -35,11 +31,6 @@ class UserNote(Base):
comment = relationship("Comment", back_populates="notes")
post = relationship("Submission", back_populates="notes")
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs:
kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"

43
files/classes/views.py
View file

@ -1,11 +1,14 @@
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from files.helpers.lazy import *
import time
class ViewerRelationship(Base):
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from files.helpers.lazy import lazy
from files.helpers.time import format_age
class ViewerRelationship(Base):
__tablename__ = "viewers"
user_id = Column(Integer, ForeignKey('users.id'), primary_key=True)
@ -17,7 +20,6 @@ class ViewerRelationship(Base):
viewer = relationship("User", primaryjoin="ViewerRelationship.viewer_id == User.id", viewonly=True)
def __init__(self, **kwargs):
if 'last_view_utc' not in kwargs:
kwargs['last_view_utc'] = int(time.time())
@ -26,36 +28,9 @@ class ViewerRelationship(Base):
@property
@lazy
def last_view_since(self):
return int(time.time()) - self.last_view_utc
@property
@lazy
def last_view_string(self):
age = self.last_view_since
if age < 60:
return "just now"
elif age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
elif age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
elif age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(self.created_utc)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if now.tm_mday < ctd.tm_mday:
months -= 1
if months < 12:
return f"{months}mo ago"
else:
years = int(months / 12)
return f"{years}yr ago"
return format_age(self.last_view_utc)

1
files/classes/volunteer_janitor.py
View file

@ -14,7 +14,6 @@ class VolunteerJanitorResult(enum.Enum):
Ban = 6
class VolunteerJanitorRecord(Base):
__tablename__ = "volunteer_janitor"
id = Column(Integer, primary_key=True)

22
files/classes/votes.py
View file

@ -1,12 +1,11 @@
from flask import *
from sqlalchemy import *
from sqlalchemy.orm import relationship
from files.classes.base import Base
from files.classes.base import CreatedBase
from files.helpers.lazy import lazy
import time
class Vote(Base):
class Vote(CreatedBase):
__tablename__ = "votes"
submission_id = Column(Integer, ForeignKey("submissions.id"), primary_key=True)
@ -14,7 +13,6 @@ class Vote(Base):
vote_type = Column(Integer, nullable=False)
app_id = Column(Integer, ForeignKey("oauth_apps.id"))
real = Column(Boolean, default=True, nullable=False)
created_utc = Column(Integer, nullable=False)
Index('votes_type_index', vote_type)
Index('vote_user_index', user_id)
@ -22,10 +20,6 @@ class Vote(Base):
user = relationship("User", lazy="subquery", viewonly=True)
post = relationship("Submission", lazy="subquery", viewonly=True)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
@ -45,12 +39,10 @@ class Vote(Base):
data=self.json_core
data["user"]=self.user.json_core
data["post"]=self.post.json_core
return data
class CommentVote(Base):
class CommentVote(CreatedBase):
__tablename__ = "commentvotes"
comment_id = Column(Integer, ForeignKey("comments.id"), primary_key=True)
@ -58,7 +50,6 @@ class CommentVote(Base):
vote_type = Column(Integer, nullable=False)
app_id = Column(Integer, ForeignKey("oauth_apps.id"))
real = Column(Boolean, default=True, nullable=False)
created_utc = Column(Integer, nullable=False)
Index('cvote_user_index', user_id)
Index('commentvotes_comments_type_index', vote_type)
@ -66,10 +57,6 @@ class CommentVote(Base):
user = relationship("User", lazy="subquery")
comment = relationship("Comment", lazy="subquery", viewonly=True)
def __init__(self, *args, **kwargs):
if "created_utc" not in kwargs: kwargs["created_utc"] = int(time.time())
super().__init__(*args, **kwargs)
def __repr__(self):
return f"<{self.__class__.__name__}(id={self.id})>"
@ -89,5 +76,4 @@ class CommentVote(Base):
data=self.json_core
data["user"]=self.user.json_core
data["comment"]=self.comment.json_core
return data

6
files/cli.py
View file

@ -1,7 +1,9 @@
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy
from .__main__ import app
from .commands.seed_db import seed_db
from files.__main__ import app
from files.commands.cron import cron_app_worker
from files.commands.seed_db import seed_db
import files.classes
db = SQLAlchemy(app)

129
files/commands/cron.py Normal file
View file

@ -0,0 +1,129 @@
import contextlib
import logging
import time
from datetime import datetime, timezone
from typing import Final
from sqlalchemy.orm import scoped_session, Session
from files.__main__ import app, db_session
from files.classes.cron.tasks import (DayOfWeek, RepeatableTask,
RepeatableTaskRun, ScheduledTaskState)
CRON_SLEEP_SECONDS: Final[int] = 15
'''
How long the app will sleep for between runs. Lower values give better
resolution, but will hit the database more.
The cost of a lower value is potentially higher lock contention. A value below
`0` will raise a `ValueError` (on call to `time.sleep`). A value of `0` is
possible but not recommended.
The sleep time is not guaranteed to be exactly this value (notably, it may be
slightly longer if the system is very busy)
This value is passed to `time.sleep()`. For more information on that, see
the Python documentation: https://docs.python.org/3/library/time.html
'''
_CRON_COMMAND_NAME = "cron"
@app.cli.command(_CRON_COMMAND_NAME)
def cron_app_worker():
'''
The "worker" process task. This actually executes tasks.
'''
logging.info("Starting scheduler worker process")
while True:
try:
_run_tasks(db_session)
except Exception as e:
logging.exception(
"An unhandled exception occurred while running tasks",
exc_info=e
)
time.sleep(CRON_SLEEP_SECONDS)
@contextlib.contextmanager
def _acquire_lock_exclusive(db: Session, table: str):
'''
Acquires an exclusive lock on the table provided by the `table` parameter.
This can be used for synchronizing the state of the specified table and
making sure no readers can access it while in the critical section.
'''
# TODO: make `table` the type LiteralString once we upgrade to python 3.11
db.begin() # we want to raise an exception if there's a txn in progress
db.execute(f"LOCK TABLE {table} IN ACCESS EXCLUSIVE MODE")
try:
yield
db.commit()
except Exception:
logging.error(
"An exception occurred during an operation in a critical section. "
"A task might not occur or might be duplicated."
)
try:
db.rollback()
except:
logging.warning(
f"Failed to rollback database. The table {table} might still "
"be locked.")
raise
def _run_tasks(db_session_factory: scoped_session):
'''
Runs tasks, attempting to guarantee that a task is ran once and only once.
This uses postgres to lock the table containing our tasks at key points in
in the process (reading the tasks and writing the last updated time).
The task itself is ran outside of this context; this is so that a long
running task does not lock the entire table for its entire run, which would
for example, prevent any statistics about status from being gathered.
'''
db: Session = db_session_factory()
with _acquire_lock_exclusive(db, RepeatableTask.__tablename__):
now: datetime = datetime.now(tz=timezone.utc)
tasks: list[RepeatableTask] = db.query(RepeatableTask).filter(
RepeatableTask.enabled == True,
RepeatableTask.frequency_day != int(DayOfWeek.NONE),
RepeatableTask.run_state != int(ScheduledTaskState.RUNNING),
(RepeatableTask.run_time_last <= now)
| (RepeatableTask.run_time_last == None),
).all()
# SQLA needs to query again for the inherited object info anyway
# so it's fine that objects in the list get expired on txn end.
# Prefer more queries to risk of task run duplication.
tasks_to_run: list[RepeatableTask] = list(filter(
lambda task: task.can_run(now), tasks))
for task in tasks_to_run:
now = datetime.now(tz=timezone.utc)
with _acquire_lock_exclusive(db, RepeatableTask.__tablename__):
# We need to check for runnability again because we don't mutex
# the RepeatableTask.run_state until now.
if not task.can_run(now):
continue
task.run_time_last = now
task.run_state_enum = ScheduledTaskState.RUNNING
db.begin()
run: RepeatableTaskRun = task.run(db, task.run_time_last_or_created_utc)
if run.exception:
# TODO: collect errors somewhere other than just here and in the
# task run object itself (see #220).
logging.exception(
f"Exception running task (ID {run.task_id}, run ID {run.id})",
exc_info=run.exception
)
db.rollback()
else:
db.commit()
with _acquire_lock_exclusive(db, RepeatableTask.__tablename__):
task.run_state_enum = ScheduledTaskState.WAITING

9
files/commands/seed_db.py
View file

@ -1,15 +1,16 @@
import hashlib
import math
from typing import Optional
import sqlalchemy
from sqlalchemy.orm import scoped_session
import sqlalchemy
from sqlalchemy.orm import Session
from werkzeug.security import generate_password_hash
from files.__main__ import app, db_session
from files.classes import User, Submission, Comment, Vote, CommentVote
from files.classes import Comment, CommentVote, Submission, User, Vote
from files.helpers.comments import bulk_recompute_descendant_counts
@app.cli.command('seed_db')
def seed_db():
seed_db_worker()
@ -23,7 +24,7 @@ def seed_db_worker(num_users = 900, num_posts = 40, num_toplevel_comments = 1000
COMMENT_UPVOTE_PROB = 0.0008
COMMENT_DOWNVOTE_PROB = 0.0003
db: scoped_session = db_session()
db: Session = db_session()
def detrand():
detrand.randstate = bytes(hashlib.sha256(detrand.randstate).hexdigest(), 'utf-8')

37
files/helpers/alerts.py
View file

@ -1,7 +1,8 @@
from files.classes import *
from flask import g
from .sanitize import *
from .const import *
from .config.const import *
def create_comment(text_html, autojanny=False):
if autojanny: author_id = AUTOJANNY_ID
@ -19,7 +20,6 @@ def create_comment(text_html, autojanny=False):
return new_comment.id
def send_repeatable_notification(uid, text, autojanny=False):
if autojanny: author_id = AUTOJANNY_ID
else: author_id = NOTIFICATIONS_ID
@ -38,13 +38,11 @@ def send_repeatable_notification(uid, text, autojanny=False):
def send_notification(uid, text, autojanny=False):
cid = notif_comment(text, autojanny)
add_notif(cid, uid)
def notif_comment(text, autojanny=False):
if autojanny:
author_id = AUTOJANNY_ID
alert = True
@ -61,7 +59,6 @@ def notif_comment(text, autojanny=False):
def notif_comment2(p):
search_html = f'%</a> has mentioned you: <a href="/post/{p.id}">%'
existing = g.db.query(Comment.id).filter(Comment.author_id == NOTIFICATIONS_ID, Comment.parent_submission == None, Comment.body_html.like(search_html)).first()
@ -81,11 +78,12 @@ def add_notif(cid, uid):
g.db.add(notif)
def NOTIFY_USERS(text, v):
def NOTIFY_USERS(text, v) -> set[int]:
notify_users = set()
for word, id in NOTIFIED_USERS.items():
if id == 0 or v.id == id: continue
if word in text.lower() and id not in notify_users: notify_users.add(id)
if word in text.lower() and id not in notify_users:
notify_users.add(id)
captured = []
for i in mention_regex.finditer(text):
@ -95,6 +93,29 @@ def NOTIFY_USERS(text, v):
captured.append(i.group(0))
user = get_user(i.group(2), graceful=True)
if user and v.id != user.id and not v.any_block_exists(user): notify_users.add(user.id)
if user and v.id != user.id and not v.any_block_exists(user):
notify_users.add(user.id)
return notify_users
def notify_submission_publish(target: Submission):
# Username mentions in title & body
text: str = f'{target.title} {target.body}'
notify_users = NOTIFY_USERS(text, target.author)
if notify_users:
comment_id = notif_comment2(target)
for user_id in notify_users:
add_notif(comment_id, user_id)
# Submission author followers
if target.author.followers:
message: str = (
f"@{target.author.username} has made a new post: "
f"[{target.title}]({target.shortlink})"
)
if target.sub:
message += f" in <a href='/h/{target.sub}'>/h/{target.sub}"
cid = notif_comment(message, autojanny=True)
for follow in target.author.followers:
add_notif(cid, follow.user_id)

25
files/helpers/caching.py Normal file
View file

@ -0,0 +1,25 @@
from files.__main__ import cache
import files.helpers.listing as listing
# i hate this.
#
# we should probably come up with a better way for this in the future.
# flask_caching is kinda weird in that it requires you to use a function
# reference to deleted a memoized function, which basically means fitting your
# code to flask_caching's worldview. it's very much not ideal and ideally would
# be less coupled in the future.
#
# the question is whether it's worth it.
def invalidate_cache(*, frontlist=False, userpagelisting=False, changeloglist=False):
'''
Invalidates the caches for the front page listing, user page listings,
and optionally, the changelog listing.
:param frontlist: Whether to invalidate the `frontlist` cache.
:param userpagelisting: Whether to invalidate the `userpagelisting` cache.
:param changeloglist: Whether to invalidate the `changeloglist` cache.
'''
if frontlist: cache.delete_memoized(listing.frontlist)
if userpagelisting: cache.delete_memoized(listing.userpagelisting)
if changeloglist: cache.delete_memoized(listing.changeloglist)

19
files/helpers/comments.py
View file

@ -1,15 +1,18 @@
from sys import stdout
from typing import Optional
import gevent
from flask import g, request
from pusher_push_notifications import PushNotifications
from sqlalchemy import select, update
from sqlalchemy.orm import Query, aliased
from sqlalchemy.sql.expression import alias, func, text
from files.classes import Comment, Notification, Subscription, User
from files.helpers.alerts import NOTIFY_USERS
from files.helpers.const import PUSHER_ID, PUSHER_KEY, SITE_ID, SITE_FULL
from files.helpers.assetcache import assetcache_path
from flask import g
from sqlalchemy import select, update
from sqlalchemy.sql.expression import func, text, alias
from sqlalchemy.orm import Query, aliased
from sys import stdout
import gevent
from typing import Optional
from files.helpers.config.environment import (PUSHER_ID, PUSHER_KEY, SITE_FULL,
SITE_ID)
if PUSHER_ID != 'blahblahblah':
beams_client = PushNotifications(instance_id=PUSHER_ID, secret_key=PUSHER_KEY)

152
files/helpers/const.py → files/helpers/config/const.py
View file

@ -1,19 +1,47 @@
import re
import sys
from copy import deepcopy
from os import environ
from enum import IntEnum
from typing import Final
from flask import request
from files.__main__ import db_session
from files.classes.sub import Sub
from files.classes.marsey import Marsey
SITE = environ.get("DOMAIN", '').strip()
SITE_ID = environ.get("SITE_ID", '').strip()
SITE_TITLE = environ.get("SITE_TITLE", '').strip()
SCHEME = environ.get('SCHEME', 'http' if 'localhost' in SITE else 'https')
SITE_FULL = SCHEME + '://' + SITE
class Service(IntEnum):
'''
An enumeration of services provided by this application
'''
THEMOTTE = 0
'''
TheMotte web application. Handles most routes and tasks performed,
including all non-chat web requests.
'''
CRON = 1
'''
Runs tasks periodicially on a set schedule
'''
CHAT = 2
'''
Chat application.
'''
MIGRATION = 3
'''
Migration mode. Used for performing database migrations
'''
@classmethod
def from_argv(cls):
if "db" in sys.argv:
return cls.MIGRATION
if "cron" in sys.argv:
return cls.CRON
if "load_chat" in sys.argv:
return cls.CHAT
return cls.THEMOTTE
@property
def enable_services(self) -> bool:
return self not in {self.CRON, self.MIGRATION}
CC = "COUNTRY CLUB"
CC_TITLE = CC.title()
@ -32,7 +60,6 @@ BASEDBOT_ID = 0
GIFT_NOTIF_ID = 9
OWNER_ID = 9
BUG_THREAD = 0
WELCOME_MSG = f"Welcome to {SITE_TITLE}! Please read [the rules](/rules) first. Then [read some of our current conversations](/) and feel free to comment or post!\n\nWe encourage people to comment even if they aren't sure they fit in; as long as your comment follows [community rules](/rules), we are happy to have posters from all backgrounds, education levels, and specialties."
ROLES={}
LEADERBOARD_LIMIT: Final[int] = 25
@ -54,11 +81,16 @@ SORTS_POSTS = {
SORTS_POSTS.update(SORTS_COMMON)
SORTS_COMMENTS = SORTS_COMMON
PUSHER_ID = environ.get("PUSHER_ID", "").strip()
PUSHER_KEY = environ.get("PUSHER_KEY", "").strip()
DEFAULT_COLOR = environ.get("DEFAULT_COLOR", "fff").strip()
COLORS = {'ff66ac','805ad5','62ca56','38a169','80ffff','2a96f3','eb4963','ff0000','f39731','30409f','3e98a7','e4432d','7b9ae4','ec72de','7f8fa6', 'f8db58','8cdbe6', DEFAULT_COLOR}
MAX_CONTENT_LENGTH = 16 * 1024 * 1024
SESSION_COOKIE_SAMESITE = "Lax"
PERMANENT_SESSION_LIFETIME = 60 * 60 * 24 * 365
DEFAULT_THEME = "TheMotte"
FORCE_HTTPS = 1
COLORS = {'ff66ac','805ad5','62ca56','38a169','80ffff','2a96f3','eb4963','ff0000','f39731','30409f','3e98a7','e4432d','7b9ae4','ec72de','7f8fa6', 'f8db58','8cdbe6', 'fff'}
SUBMISSION_FLAIR_LENGTH_MAXIMUM: Final[int] = 350
SUBMISSION_TITLE_LENGTH_MAXIMUM: Final[int] = 500
SUBMISSION_URL_LENGTH_MAXIMUM: Final[int] = 2048
SUBMISSION_BODY_LENGTH_MAXIMUM: Final[int] = 20000
COMMENT_BODY_LENGTH_MAXIMUM: Final[int] = 10000
MESSAGE_BODY_LENGTH_MAXIMUM: Final[int] = 10000
@ -72,6 +104,7 @@ ERROR_MESSAGES = {
405: "Something went wrong and it's probably my fault. If you can do it reliably, or it's causing problems for you, please report it!",
409: "There's a conflict between what you're trying to do and what you or someone else has done and because of that you can't do what you're trying to do.",
413: "Max file size is 8 MB",
415: "That file type isn't allowed to be uploaded here",
422: "Something is wrong about your request. If you keep getting this unexpectedly, please report it!",
429: "Are you hammering the site? Stop that, yo.",
500: "Something went wrong and it's probably my fault. If you can do it reliably, or it's causing problems for you, please report it!",
@ -98,6 +131,7 @@ WERKZEUG_ERROR_DESCRIPTIONS = {
}
IMAGE_FORMATS = ['png','gif','jpg','jpeg','webp']
IMAGE_URL_ENDINGS = IMAGE_FORMATS + ['.webp', '.jpg', '.png', '.gif', '.jpeg', '?maxwidth=9999', '&fidelity=high']
VIDEO_FORMATS = ['mp4','webm','mov','avi','mkv','flv','m4v','3gp']
AUDIO_FORMATS = ['mp3','wav','ogg','aac','m4a','flac']
NO_TITLE_EXTENSIONS = IMAGE_FORMATS + VIDEO_FORMATS + AUDIO_FORMATS
@ -108,11 +142,15 @@ FEATURES = {
PERMS = {
"DEBUG_LOGIN_TO_OTHERS": 3,
'PERFORMANCE_KILL_PROCESS': 3,
'PERFORMANCE_SCALE_UP_DOWN': 3,
'PERFORMANCE_RELOAD': 3,
'PERFORMANCE_STATS': 3,
"PERFORMANCE_KILL_PROCESS": 3,
"PERFORMANCE_SCALE_UP_DOWN": 3,
"PERFORMANCE_RELOAD": 3,
"PERFORMANCE_STATS": 3,
"POST_COMMENT_MODERATION": 2,
"POST_EDITING": 3,
"SCHEDULER": 2,
"SCHEDULER_POSTS": 2,
"SCHEDULER_TASK_TRACEBACK": 3,
"USER_SHADOWBAN": 2,
}
@ -137,82 +175,6 @@ NOTIFIED_USERS = {
patron = 'Patron'
discounts = {
69: 0.02,
70: 0.04,
71: 0.06,
72: 0.08,
73: 0.10,
}
CF_KEY = environ.get("CF_KEY", "").strip()
CF_ZONE = environ.get("CF_ZONE", "").strip()
CF_HEADERS = {"Authorization": f"Bearer {CF_KEY}", "Content-Type": "application/json"}
dues = int(environ.get("DUES").strip())
christian_emojis = (':#marseyjesus:',':#marseyimmaculate:',':#marseymothermary:',':#marseyfatherjoseph:',':#gigachadorthodox:',':#marseyorthodox:',':#marseyorthodoxpat:')
db = db_session()
marseys_const = [x[0] for x in db.query(Marsey.name).filter(Marsey.name!='chudsey').all()]
marseys_const2 = marseys_const + ['chudsey','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z','0','1','2','3','4','5','6','7','8','9','exclamationpoint','period','questionmark']
db.close()
valid_username_chars = 'a-zA-Z0-9_\\-'
valid_username_regex = re.compile("^[a-zA-Z0-9_\\-]{3,25}$", flags=re.A)
mention_regex = re.compile('(^|\\s|<p>)@(([a-zA-Z0-9_\\-]){1,25})', flags=re.A)
mention_regex2 = re.compile('<p>@(([a-zA-Z0-9_\\-]){1,25})', flags=re.A)
valid_password_regex = re.compile("^.{8,100}$", flags=re.A)
marsey_regex = re.compile("[a-z0-9]{1,30}", flags=re.A)
tags_regex = re.compile("[a-z0-9: ]{1,200}", flags=re.A)
valid_sub_regex = re.compile("^[a-zA-Z0-9_\\-]{3,20}$", flags=re.A)
query_regex = re.compile("(\\w+):(\\S+)", flags=re.A)
title_regex = re.compile("[^\\w ]", flags=re.A)
based_regex = re.compile("based and (.{1,20}?)(-| )pilled", flags=re.I|re.A)
controversial_regex = re.compile('["> ](https:\\/\\/old\\.reddit\\.com/r/[a-zA-Z0-9_]{3,20}\\/comments\\/[\\w\\-.#&/=\\?@%+]{5,250})["< ]', flags=re.A)
fishylinks_regex = re.compile("https?://\\S+", flags=re.A)
spoiler_regex = re.compile('''\\|\\|(.+)\\|\\|''', flags=re.A)
reddit_regex = re.compile('(^|\\s|<p>)\\/?((r|u)\\/(\\w|-){3,25})(?![^<]*<\\/(code|pre|a)>)', flags=re.A)
sub_regex = re.compile('(^|\\s|<p>)\\/?(h\\/(\\w|-){3,25})', flags=re.A)
# Bytes that shouldn't be allowed in user-submitted text
# U+200E is LTR toggle, U+200F is RTL toggle, U+200B and U+FEFF are Zero-Width Spaces,
# and U+1242A is a massive and terrifying cuneiform numeral
unwanted_bytes_regex = re.compile("\u200e|\u200f|\u200b|\ufeff|\U0001242a")
whitespace_regex = re.compile('\\s+')
strikethrough_regex = re.compile('''~{1,2}([^~]+)~{1,2}''', flags=re.A)
mute_regex = re.compile("/mute @([a-z0-9_\\-]{3,25}) ([0-9])+", flags=re.A)
emoji_regex = re.compile(f"[^a]>\\s*(:[!#@]{{0,3}}[{valid_username_chars}]+:\\s*)+<\\/", flags=re.A)
emoji_regex2 = re.compile(f"(?<!\"):([!#@{valid_username_chars}]{{1,31}}?):", flags=re.A)
emoji_regex3 = re.compile(f"(?<!\"):([!@{valid_username_chars}]{{1,31}}?):", flags=re.A)
snappy_url_regex = re.compile('<a href=\\"(https?:\\/\\/[a-z]{1,20}\\.[\\w:~,()\\-.#&\\/=?@%;+]{5,250})\\" rel=\\"nofollow noopener noreferrer\\" target=\\"_blank\\">([\\w:~,()\\-.#&\\/=?@%;+]{5,250})<\\/a>', flags=re.A)
# Technically this allows stuff that is not a valid email address, but realistically
# we care "does this email go to the correct person" rather than "is this email
# address syntactically valid", so if we care we should be sending a confirmation
# link, and otherwise should be pretty liberal in what we accept here.
email_regex = re.compile('[^@]+@[^@]+\\.[^@]+', flags=re.A)
utm_regex = re.compile('utm_[a-z]+=[a-z0-9_]+&', flags=re.A)
utm_regex2 = re.compile('[?&]utm_[a-z]+=[a-z0-9_]+', flags=re.A)
YOUTUBE_KEY = environ.get("YOUTUBE_KEY", "").strip()
proxies = {}
approved_embed_hosts = [
@ -276,6 +238,6 @@ video_sub_regex = re.compile(f'(<p>[^<]*)(https:\\/\\/([a-z0-9-]+\\.)*({hosts})\
procoins_li = (0,2500,5000,10000,25000,50000,125000,250000)
from files.helpers.regex import *
from files.helpers.config.regex import *
def make_name(*args, **kwargs): return request.base_url

109
files/helpers/config/environment.py Normal file
View file

@ -0,0 +1,109 @@
'''
Environment data. Please don't use `files.helpers.config.const` for things that
aren't constants. If it's an environment configuration, it should go in here.
'''
from os import environ
from files.helpers.strings import bool_from_string
SITE = environ.get("DOMAIN", '').strip()
SITE_ID = environ.get("SITE_ID").strip()
SITE_TITLE = environ.get("SITE_TITLE").strip()
SCHEME = environ.get('SCHEME', 'http' if 'localhost' in SITE else 'https')
if "localhost" in SITE:
SITE_FULL = 'http://' + SITE
else:
SITE_FULL = 'https://' + SITE
WELCOME_MSG = (
f"Welcome to {SITE_TITLE}! Please read [the rules](/rules) first. "
"Then [read some of our current conversations](/) and feel free to comment "
"or post!\n"
"We encourage people to comment even if they aren't sure they fit in; as "
"long as your comment follows [community rules](/rules), we are happy to "
"have posters from all backgrounds, education levels, and specialties."
)
SQLALCHEMY_TRACK_MODIFICATIONS = False
DATABASE_URL = environ.get("DATABASE_URL", "postgresql://postgres@localhost:5432")
SECRET_KEY = environ.get('MASTER_KEY', '')
SERVER_NAME = environ.get("DOMAIN").strip()
SESSION_COOKIE_SECURE = "localhost" not in SERVER_NAME
DEFAULT_COLOR = environ.get("DEFAULT_COLOR", "fff").strip()
DEFAULT_TIME_FILTER = environ.get("DEFAULT_TIME_FILTER", "all").strip()
HCAPTCHA_SITEKEY = environ.get("HCAPTCHA_SITEKEY","").strip()
HCAPTCHA_SECRET = environ.get("HCAPTCHA_SECRET","").strip()
if not SECRET_KEY:
raise Exception("Secret key not set!")
# spam filter
SPAM_SIMILARITY_THRESHOLD = float(environ.get("SPAM_SIMILARITY_THRESHOLD", 0.5))
''' Spam filter similarity threshold (posts) '''
SPAM_URL_SIMILARITY_THRESHOLD = float(environ.get("SPAM_URL_SIMILARITY_THRESHOLD", 0.1))
''' Spam filter similarity threshold for URLs (posts) '''
SPAM_SIMILAR_COUNT_THRESHOLD = int(environ.get("SPAM_SIMILAR_COUNT_THRESHOLD", 10))
''' Spam filter similarity count (posts) '''
COMMENT_SPAM_SIMILAR_THRESHOLD = float(environ.get("COMMENT_SPAM_SIMILAR_THRESHOLD", 0.5))
''' Spam filter similarity threshold (comments)'''
COMMENT_SPAM_COUNT_THRESHOLD = int(environ.get("COMMENT_SPAM_COUNT_THRESHOLD", 10))
''' Spam filter similarity count (comments) '''
CACHE_REDIS_URL = environ.get("REDIS_URL", "redis://localhost")
MAIL_SERVER = environ.get("MAIL_SERVER", "").strip()
MAIL_PORT = 587
MAIL_USE_TLS = True
MAIL_USERNAME = environ.get("MAIL_USERNAME", "").strip()
MAIL_PASSWORD = environ.get("MAIL_PASSWORD", "").strip()
DESCRIPTION = environ.get("DESCRIPTION", "DESCRIPTION GOES HERE").strip()
SQLALCHEMY_DATABASE_URI = DATABASE_URL
MENTION_LIMIT = int(environ.get('MENTION_LIMIT', 100))
''' Maximum amount of username mentions '''
MULTIMEDIA_EMBEDDING_ENABLED = bool_from_string(environ.get('MULTIMEDIA_EMBEDDING_ENABLED', False))
'''
Whether multimedia will be embedded into a page. Note that this does not
affect posts or comments retroactively.
'''
RESULTS_PER_PAGE_COMMENTS = int(environ.get('RESULTS_PER_PAGE_COMMENTS', 50))
SCORE_HIDING_TIME_HOURS = int(environ.get('SCORE_HIDING_TIME_HOURS', 0))
ENABLE_SERVICES = bool_from_string(environ.get('ENABLE_SERVICES', False))
'''
Whether to start up deferred tasks. Usually `True` when running as an app and
`False` when running as a script (for example to perform migrations).
See https://github.com/themotte/rDrama/pull/427 for more info.
'''
DBG_VOLUNTEER_PERMISSIVE = bool_from_string(environ.get('DBG_VOLUNTEER_PERMISSIVE', False))
VOLUNTEER_JANITOR_ENABLE = bool_from_string(environ.get('VOLUNTEER_JANITOR_ENABLE', True))
RATE_LIMITER_ENABLED = not bool_from_string(environ.get('DBG_LIMITER_DISABLED', False))
ENABLE_DOWNVOTES = not bool_from_string(environ.get('DISABLE_DOWNVOTES', False))
CARD_VIEW = bool_from_string(environ.get("CARD_VIEW", True))
FINGERPRINT_TOKEN = environ.get("FP", None)
# other stuff from const.py that aren't constants
CLUB_TRUESCORE_MINIMUM = int(environ.get("DUES").strip())
IMGUR_KEY = environ.get("IMGUR_KEY", "").strip()
PUSHER_ID = environ.get("PUSHER_ID", "").strip()
PUSHER_KEY = environ.get("PUSHER_KEY", "").strip()
YOUTUBE_KEY = environ.get("YOUTUBE_KEY", "").strip()
CF_KEY = environ.get("CF_KEY", "").strip()
CF_ZONE = environ.get("CF_ZONE", "").strip()
CF_HEADERS = {
"Authorization": f"Bearer {CF_KEY}",
"Content-Type": "application/json"
}

80
files/helpers/config/regex.py Normal file
View file

@ -0,0 +1,80 @@
import re
# usernames
valid_username_chars = 'a-zA-Z0-9_\\-'
valid_username_regex = re.compile("^[a-zA-Z0-9_\\-]{3,25}$", flags=re.A)
mention_regex = re.compile('(^|\\s|<p>)@(([a-zA-Z0-9_\\-]){1,25})', flags=re.A)
mention_regex2 = re.compile('<p>@(([a-zA-Z0-9_\\-]){1,25})', flags=re.A)
valid_password_regex = re.compile("^.{8,100}$", flags=re.A)
marsey_regex = re.compile("[a-z0-9]{1,30}", flags=re.A)
tags_regex = re.compile("[a-z0-9: ]{1,200}", flags=re.A)
valid_sub_regex = re.compile("^[a-zA-Z0-9_\\-]{3,20}$", flags=re.A)
query_regex = re.compile("(\\w+):(\\S+)", flags=re.A)
title_regex = re.compile("[^\\w ]", flags=re.A)
based_regex = re.compile("based and (.{1,20}?)(-| )pilled", flags=re.I|re.A)
controversial_regex = re.compile('["> ](https:\\/\\/old\\.reddit\\.com/r/[a-zA-Z0-9_]{3,20}\\/comments\\/[\\w\\-.#&/=\\?@%+]{5,250})["< ]', flags=re.A)
fishylinks_regex = re.compile("https?://\\S+", flags=re.A)
spoiler_regex = re.compile('''\\|\\|(.+)\\|\\|''', flags=re.A)
reddit_regex = re.compile('(^|\\s|<p>)\\/?((r|u)\\/(\\w|-){3,25})(?![^<]*<\\/(code|pre|a)>)', flags=re.A)
sub_regex = re.compile('(^|\\s|<p>)\\/?(h\\/(\\w|-){3,25})', flags=re.A)
unwanted_bytes_regex = re.compile("\u200e|\u200f|\u200b|\ufeff|\U0001242a")
'''
Bytes that shouldn't be allowed in user-submitted text
U+200E is LTR toggle, U+200F is RTL toggle, U+200B and U+FEFF are Zero-Width
Spaces, and U+1242A is a massive and terrifying cuneiform numeral
'''
whitespace_regex = re.compile('\\s+')
strikethrough_regex = re.compile('''~{1,2}([^~]+)~{1,2}''', flags=re.A)
mute_regex = re.compile("/mute @([a-z0-9_\\-]{3,25}) ([0-9])+", flags=re.A)
emoji_regex = re.compile(f"[^a]>\\s*(:[!#@]{{0,3}}[{valid_username_chars}]+:\\s*)+<\\/", flags=re.A)
emoji_regex2 = re.compile(f"(?<!\"):([!#@{valid_username_chars}]{{1,31}}?):", flags=re.A)
emoji_regex3 = re.compile(f"(?<!\"):([!@{valid_username_chars}]{{1,31}}?):", flags=re.A)
snappy_url_regex = re.compile('<a href=\\"(https?:\\/\\/[a-z]{1,20}\\.[\\w:~,()\\-.#&\\/=?@%;+]{5,250})\\" rel=\\"nofollow noopener noreferrer\\" target=\\"_blank\\">([\\w:~,()\\-.#&\\/=?@%;+]{5,250})<\\/a>', flags=re.A)
email_regex = re.compile('[^@]+@[^@]+\\.[^@]+', flags=re.A)
'''
Regex to use for email addresses.
.. note::
Technically this allows stuff that is not a valid email address, but
realistically we care "does this email go to the correct person" rather
than "is this email address syntactically valid", so if we care we should
be sending a confirmation link, and otherwise should be pretty liberal in
what we accept here.
'''
utm_regex = re.compile('utm_[a-z]+=[a-z0-9_]+&', flags=re.A)
utm_regex2 = re.compile('[?&]utm_[a-z]+=[a-z0-9_]+', flags=re.A)
# urls
youtube_regex = re.compile('(<p>[^<]*)(https:\\/\\/youtube\\.com\\/watch\\?v\\=([a-z0-9-_]{5,20})[\\w\\-.#&/=\\?@%+]*)', flags=re.I|re.A)
yt_id_regex = re.compile('[a-z0-9-_]{5,20}', flags=re.I|re.A)
image_regex = re.compile("(^|\\s)(https:\\/\\/[\\w\\-.#&/=\\?@%;+]{5,250}(\\.png|\\.jpg|\\.jpeg|\\.gif|\\.webp|maxwidth=9999|fidelity=high))($|\\s)", flags=re.I|re.A)
linefeeds_regex = re.compile("([^\\n])\\n([^\\n])", flags=re.A)
html_title_regex = re.compile(r"<title>(.{1,200})</title>", flags=re.I)
css_url_regex = re.compile(r'url\(\s*[\'"]?(.*?)[\'"]?\s*\)', flags=re.I|re.A)

130
files/helpers/content.py
View file

@ -1,16 +1,92 @@
from typing import Any, TYPE_CHECKING, Optional, Union
from __future__ import annotations
from files.helpers.const import PERMS
import random
import urllib.parse
from typing import TYPE_CHECKING, Optional, Union
from sqlalchemy.orm import Session
from files.helpers.config.const import PERMS
if TYPE_CHECKING:
from files.classes import Submission, Comment, User
else:
Submission = Any
Comment = Any
User = Any
from files.classes import Comment, Submission, User
Submittable = Union[Submission, Comment]
def moderated_body(target:Union[Submission, Comment],
v:Optional[User]) -> Optional[str]:
def _replace_urls(url:str) -> str:
def _replace_extensions(url:str, exts:list[str]) -> str:
for ext in exts:
url = url.replace(f'.{ext}', '.webp')
return url
for rd in ("://reddit.com", "://new.reddit.com", "://www.reddit.com", "://redd.it", "://libredd.it", "://teddit.net"):
url = url.replace(rd, "://old.reddit.com")
url = url.replace("nitter.net", "twitter.com") \
.replace("old.reddit.com/gallery", "reddit.com/gallery") \
.replace("https://youtu.be/", "https://youtube.com/watch?v=") \
.replace("https://music.youtube.com/watch?v=", "https://youtube.com/watch?v=") \
.replace("https://streamable.com/", "https://streamable.com/e/") \
.replace("https://youtube.com/shorts/", "https://youtube.com/watch?v=") \
.replace("https://mobile.twitter", "https://twitter") \
.replace("https://m.facebook", "https://facebook") \
.replace("m.wikipedia.org", "wikipedia.org") \
.replace("https://m.youtube", "https://youtube") \
.replace("https://www.youtube", "https://youtube") \
.replace("https://www.twitter", "https://twitter") \
.replace("https://www.instagram", "https://instagram") \
.replace("https://www.tiktok", "https://tiktok")
if "/i.imgur.com/" in url:
url = _replace_extensions(url, ['png', 'jpg', 'jpeg'])
elif "/media.giphy.com/" in url or "/c.tenor.com/" in url:
url = _replace_extensions(url, ['gif'])
elif "/i.ibb.com/" in url:
url = _replace_extensions(url, ['png', 'jpg', 'jpeg', 'gif'])
if url.startswith("https://streamable.com/") and not url.startswith("https://streamable.com/e/"):
url = url.replace("https://streamable.com/", "https://streamable.com/e/")
return url
def _httpsify_and_remove_tracking_urls(url:str) -> urllib.parse.ParseResult:
parsed_url = urllib.parse.urlparse(url)
domain = parsed_url.netloc
is_reddit_twitter_instagram_tiktok:bool = domain in \
('old.reddit.com','twitter.com','instagram.com','tiktok.com')
if is_reddit_twitter_instagram_tiktok:
query = ""
else:
qd = urllib.parse.parse_qs(parsed_url.query)
filtered = {k: val for k, val in qd.items() if not k.startswith('utm_') and not k.startswith('ref_')}
query = urllib.parse.urlencode(filtered, doseq=True)
new_url = urllib.parse.ParseResult(
scheme="https",
netloc=parsed_url.netloc,
path=parsed_url.path,
params=parsed_url.params,
query=query,
fragment=parsed_url.fragment,
)
return new_url
def canonicalize_url(url:str) -> str:
return _replace_urls(url)
def canonicalize_url2(url:str, *, httpsify:bool=False) -> urllib.parse.ParseResult:
url_parsed = _replace_urls(url)
if httpsify:
url_parsed = _httpsify_and_remove_tracking_urls(url)
else:
url_parsed = urllib.parse.urlparse(url)
return url_parsed
def moderated_body(target:Submittable, v:Optional[User]) -> Optional[str]:
if v and (v.admin_level >= PERMS['POST_COMMENT_MODERATION'] \
or v.id == target.author_id):
return None
@ -18,3 +94,39 @@ def moderated_body(target:Union[Submission, Comment],
if target.is_banned or target.filter_state == 'removed': return 'Removed'
if target.filter_state == 'filtered': return 'Filtered'
return None
def body_displayed(target:Submittable, v:Optional[User], is_html:bool) -> str:
moderated:Optional[str] = moderated_body(target, v)
if moderated: return moderated
body = target.body_html if is_html else target.body
if not body: return ""
if not v: return body
body = body.replace("old.reddit.com", v.reddit)
if v.nitter and '/i/' not in body and '/retweets' not in body:
body = body.replace("www.twitter.com", "nitter.net").replace("twitter.com", "nitter.net")
return body
def execute_shadowbanned_fake_votes(db:Session, target:Submittable, v:Optional[User]):
if not target or not v: return
if not v.shadowbanned: return
if v.id != target.author_id: return
if not (86400 > target.age_seconds > 20): return
ti = max(target.age_seconds // 60, 1)
maxupvotes = min(ti, 11)
rand = random.randint(0, maxupvotes)
if target.upvotes >= rand: return
amount = random.randint(0, 3)
if amount != 1: return
if hasattr(target, 'views'):
target.views += amount*random.randint(3, 5)
target.upvotes += amount
db.add(target)
db.commit()

2
files/helpers/contentsorting.py
View file

@ -5,7 +5,7 @@ from typing import Any, TYPE_CHECKING
from sqlalchemy.sql import func
from sqlalchemy.orm import Query
from files.helpers.const import *
from files.helpers.config.const import *
if TYPE_CHECKING:
from files.classes.comment import Comment

59
files/helpers/embeds.py Normal file
View file

@ -0,0 +1,59 @@
'''
Assists with adding embeds to submissions.
This module is not intended to be imported using the `from X import Y` syntax.
Example usage:
```py
import files.helpers.embeds as embeds
embeds.youtube("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
```
'''
import urllib.parse
from typing import Optional
import requests
from files.helpers.config.environment import YOUTUBE_KEY
from files.helpers.config.regex import yt_id_regex
__all__ = ('twitter', 'youtube',)
def twitter(url:str) -> Optional[str]:
try:
return requests.get(
url="https://publish.twitter.com/oembed",
params={"url":url, "omit_script":"t"}, timeout=5).json()["html"]
except:
return None
def youtube(url:str) -> Optional[str]:
if not YOUTUBE_KEY: return None
url = urllib.parse.unquote(url).replace('?t', '&t')
yt_id = url.split('https://youtube.com/watch?v=')[1].split('&')[0].split('%')[0]
if not yt_id_regex.fullmatch(yt_id): return None
try:
req = requests.get(
url=f"https://www.googleapis.com/youtube/v3/videos?id={yt_id}&key={YOUTUBE_KEY}&part=contentDetails",
timeout=5).json()
except:
return None
if not req.get('items'): return None
params = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
t = params.get('t', params.get('start', [0]))[0]
if isinstance(t, str): t = t.replace('s','')
embed = f'<lite-youtube videoid="{yt_id}" params="autoplay=1&modestbranding=1'
if t:
try:
embed += f'&start={int(t)}'
except:
pass
embed += '"></lite-youtube>'
return embed

20
files/helpers/get.py
View file

@ -1,12 +1,14 @@
from __future__ import annotations
from collections import defaultdict
from typing import Callable, Iterable, List, Optional, Type, Union
from flask import g
from flask import abort, g
from sqlalchemy import and_, or_, func
from sqlalchemy.orm import Query, scoped_session, selectinload
from files.classes import *
from files.helpers.const import AUTOJANNY_ID
from files.helpers.config.const import AUTOJANNY_ID
from files.helpers.contentsorting import sort_comment_results
@ -78,20 +80,22 @@ def get_account(
id:Union[str,int],
v:Optional[User]=None,
graceful:bool=False,
include_blocks:bool=False) -> Optional[User]:
include_blocks:bool=False,
db:Optional[scoped_session]=None) -> Optional[User]:
try:
id = int(id)
except:
if graceful: return None
abort(404)
user = g.db.get(User, id)
if not db: db = g.db
user = db.get(User, id)
if not user:
if graceful: return None
abort(404)
if v and include_blocks:
user = _add_block_props(user, v)
user = _add_block_props(user, v, db)
return user
@ -387,8 +391,10 @@ def get_domain(s:str) -> Optional[BannedDomain]:
def _add_block_props(
target:Union[Submission, Comment, User],
v:Optional[User]):
v:Optional[User],
db:Optional[scoped_session]=None):
if not v: return target
if not db: db = g.db
id = None
if any(isinstance(target, cls) for cls in [Submission, Comment]):
@ -408,7 +414,7 @@ def _add_block_props(
target.is_blocked = False
return target
block = g.db.query(UserBlock).filter(
block = db.query(UserBlock).filter(
or_(
and_(
UserBlock.user_id == v.id,

53
files/helpers/jinja2.py
View file

@ -1,13 +1,19 @@
from os import listdir, environ
import random
import time
from os import listdir
from jinja2 import pass_context
from files.__main__ import app
from .get import *
from .const import *
from files.classes.cron.tasks import ScheduledTaskType
from files.helpers.assetcache import assetcache_path
from files.helpers.config.environment import (CARD_VIEW, DEFAULT_COLOR,
ENABLE_DOWNVOTES, FINGERPRINT_TOKEN, PUSHER_ID, SITE, SITE_FULL, SITE_ID,
SITE_TITLE)
from files.helpers.time import format_age, format_datetime
from .config.const import *
from .get import *
@app.template_filter("computer_size")
def computer_size(size_bytes:int) -> str:
@ -31,36 +37,15 @@ def post_embed(id, v):
if p: return render_template("submission_listing.html", listing=[p], v=v)
return ''
@app.template_filter("timestamp")
def timestamp(timestamp):
if not timestamp: return ''
return format_datetime(timestamp)
age = int(time.time()) - timestamp
if age < 60:
return "just now"
elif age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
elif age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
elif age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(timestamp)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if now.tm_mday < ctd.tm_mday:
months -= 1
if months < 12:
return f"{months}mo ago"
else:
years = int(months / 12)
return f"{years}yr ago"
@app.template_filter("agestamp")
def agestamp(timestamp):
if not timestamp: return ''
return format_age(timestamp)
@app.template_filter("asset")
@ -71,7 +56,6 @@ def template_asset(asset_path):
@app.context_processor
def inject_constants():
return {
"environ":environ,
"SITE":SITE,
"SITE_ID":SITE_ID,
"SITE_TITLE":SITE_TITLE,
@ -84,8 +68,12 @@ def inject_constants():
"CC_TITLE":CC_TITLE,
"listdir":listdir,
"config":app.config.get,
"ENABLE_DOWNVOTES": ENABLE_DOWNVOTES,
"CARD_VIEW": CARD_VIEW,
"FINGERPRINT_TOKEN": FINGERPRINT_TOKEN,
"COMMENT_BODY_LENGTH_MAXIMUM":COMMENT_BODY_LENGTH_MAXIMUM,
"SUBMISSION_BODY_LENGTH_MAXIMUM":SUBMISSION_BODY_LENGTH_MAXIMUM,
"SUBMISSION_TITLE_LENGTH_MAXIMUM":SUBMISSION_TITLE_LENGTH_MAXIMUM,
"DEFAULT_COLOR":DEFAULT_COLOR,
"COLORS":COLORS,
"THEMES":THEMES,
@ -95,6 +83,7 @@ def inject_constants():
"SORTS_COMMENTS":SORTS_COMMENTS,
"SORTS_POSTS":SORTS_POSTS,
"CSS_LENGTH_MAXIMUM":CSS_LENGTH_MAXIMUM,
"ScheduledTaskType":ScheduledTaskType,
}

17
files/helpers/lazy.py
View file

@ -1,18 +1,13 @@
# Prevents certain properties from having to be recomputed each time they
# are referenced
def lazy(f):
'''
Prevents certain properties from having to be recomputed each time they are
referenced
'''
def wrapper(*args, **kwargs):
o = args[0]
if "_lazy" not in o.__dict__: o.__dict__["_lazy"] = {}
if f.__name__ not in o.__dict__["_lazy"]: o.__dict__["_lazy"][f.__name__] = f(*args, **kwargs)
if f.__name__ not in o.__dict__["_lazy"]:
o.__dict__["_lazy"][f.__name__] = f(*args, **kwargs)
return o.__dict__["_lazy"][f.__name__]
wrapper.__name__ = f.__name__
return wrapper

140
files/helpers/listing.py Normal file
View file

@ -0,0 +1,140 @@
"""
Module for listings.
"""
import time
from typing import Final
from flask import g
from sqlalchemy.sql.expression import not_
from files.__main__ import cache
from files.classes.submission import Submission
from files.classes.user import User
from files.classes.votes import Vote
from files.helpers.contentsorting import apply_time_filter, sort_objects
from files.helpers.strings import sql_ilike_clean
FRONTLIST_TIMEOUT_SECS: Final[int] = 86400
USERPAGELISTING_TIMEOUT_SECS: Final[int] = 86400
CHANGELOGLIST_TIMEOUT_SECS: Final[int] = 86400
@cache.memoize(timeout=FRONTLIST_TIMEOUT_SECS)
def frontlist(v=None, sort='new', page=1, t="all", ids_only=True, ccmode="false", filter_words='', gt=0, lt=0, sub=None, site=None):
posts = g.db.query(Submission)
if v and v.hidevotedon:
voted = [x[0] for x in g.db.query(Vote.submission_id).filter_by(user_id=v.id).all()]
posts = posts.filter(Submission.id.notin_(voted))
if not v or v.admin_level < 2:
filter_clause = (Submission.filter_state != 'filtered') & (Submission.filter_state != 'removed')
if v:
filter_clause = filter_clause | (Submission.author_id == v.id)
posts = posts.filter(filter_clause)
if sub: posts = posts.filter_by(sub=sub.name)
elif v: posts = posts.filter((Submission.sub == None) | Submission.sub.notin_(v.all_blocks))
if gt: posts = posts.filter(Submission.created_utc > gt)
if lt: posts = posts.filter(Submission.created_utc < lt)
if not gt and not lt:
posts = apply_time_filter(posts, t, Submission)
if (ccmode == "true"):
posts = posts.filter(Submission.club == True)
posts = posts.filter_by(is_banned=False, private=False, deleted_utc = 0)
if ccmode == "false" and not gt and not lt:
posts = posts.filter_by(stickied=None)
if v and v.admin_level < 2:
posts = posts.filter(Submission.author_id.notin_(v.userblocks))
if not (v and v.changelogsub):
posts = posts.filter(not_(Submission.title.ilike('[changelog]%')))
if v and filter_words:
for word in filter_words:
word = sql_ilike_clean(word).strip()
posts = posts.filter(not_(Submission.title.ilike(f'%{word}%')))
if not (v and v.shadowbanned):
posts = posts.join(User, User.id == Submission.author_id).filter(User.shadowbanned == None)
posts = sort_objects(posts, sort, Submission)
if v:
size = v.frontsize or 25
else:
size = 25
posts = posts.offset(size * (page - 1)).limit(size+1).all()
next_exists = (len(posts) > size)
posts = posts[:size]
if page == 1 and ccmode == "false" and not gt and not lt:
pins = g.db.query(Submission).filter(Submission.stickied != None, Submission.is_banned == False)
if sub: pins = pins.filter_by(sub=sub.name)
elif v:
pins = pins.filter((Submission.sub == None) | Submission.sub.notin_(v.all_blocks))
if v.admin_level < 2:
pins = pins.filter(Submission.author_id.notin_(v.userblocks))
pins = pins.all()
for pin in pins:
if pin.stickied_utc and int(time.time()) > pin.stickied_utc:
pin.stickied = None
pin.stickied_utc = None
g.db.add(pin)
pins.remove(pin)
posts = pins + posts
if ids_only: posts = [x.id for x in posts]
g.db.commit()
return posts, next_exists
@cache.memoize(timeout=USERPAGELISTING_TIMEOUT_SECS)
def userpagelisting(u:User, site=None, v=None, page=1, sort="new", t="all"):
if u.shadowbanned and not (v and (v.admin_level > 1 or v.id == u.id)): return []
posts = g.db.query(Submission.id).filter_by(author_id=u.id, is_pinned=False)
if not (v and (v.admin_level > 1 or v.id == u.id)):
posts = posts.filter_by(deleted_utc=0, is_banned=False, private=False, ghost=False)
posts = apply_time_filter(posts, t, Submission)
posts = sort_objects(posts, sort, Submission)
posts = posts.offset(25 * (page - 1)).limit(26).all()
return [x[0] for x in posts]
@cache.memoize(timeout=CHANGELOGLIST_TIMEOUT_SECS)
def changeloglist(v=None, sort="new", page=1, t="all", site=None):
posts = g.db.query(Submission.id).filter_by(is_banned=False, private=False,).filter(Submission.deleted_utc == 0)
if v.admin_level < 2:
posts = posts.filter(Submission.author_id.notin_(v.userblocks))
admins = [x[0] for x in g.db.query(User.id).filter(User.admin_level > 0).all()]
posts = posts.filter(Submission.title.ilike('_changelog%'), Submission.author_id.in_(admins))
if t != 'all':
posts = apply_time_filter(posts, t, Submission)
posts = sort_objects(posts, sort, Submission)
posts = posts.offset(25 * (page - 1)).limit(26).all()
return [x[0] for x in posts]

13
files/helpers/regex.py
View file

@ -1,13 +0,0 @@
import re
youtube_regex = re.compile('(<p>[^<]*)(https:\\/\\/youtube\\.com\\/watch\\?v\\=([a-z0-9-_]{5,20})[\\w\\-.#&/=\\?@%+]*)', flags=re.I|re.A)
yt_id_regex = re.compile('[a-z0-9-_]{5,20}', flags=re.I|re.A)
image_regex = re.compile("(^|\\s)(https:\\/\\/[\\w\\-.#&/=\\?@%;+]{5,250}(\\.png|\\.jpg|\\.jpeg|\\.gif|\\.webp|maxwidth=9999|fidelity=high))($|\\s)", flags=re.I|re.A)
linefeeds_regex = re.compile("([^\\n])\\n([^\\n])", flags=re.A)
html_title_regex = re.compile(r"<title>(.{1,200})</title>", flags=re.I)
css_url_regex = re.compile(r'url\(\s*[\'"]?(.*?)[\'"]?\s*\)', flags=re.I|re.A)

55
files/helpers/sanitize.py
View file

@ -1,20 +1,27 @@
import functools
import html
import bleach
from bs4 import BeautifulSoup
from bleach.linkifier import LinkifyFilter, build_url_re
from functools import partial
from .get import *
from os import path, environ
import re
from mistletoe import markdown
from json import loads, dump
from random import random, choice
import urllib.parse
from functools import partial
from os import path
from typing import Optional
import bleach
import gevent
import time
import requests
from files.helpers.regex import *
from files.__main__ import app
from bleach.linkifier import LinkifyFilter, build_url_re
from bs4 import BeautifulSoup
from flask import abort, g
from mistletoe import markdown
from files.classes.domains import BannedDomain
from files.classes.marsey import Marsey
from files.helpers.config.const import (embed_fullmatch_regex,
image_check_regex, video_sub_regex)
from files.helpers.config.environment import (MENTION_LIMIT,
MULTIMEDIA_EMBEDDING_ENABLED,
SITE_FULL)
from files.helpers.config.regex import *
from files.helpers.get import get_user, get_users
TLDS = ('ac','ad','ae','aero','af','ag','ai','al','am','an','ao','aq','ar',
'arpa','as','asia','at','au','aw','ax','az','ba','bb','bd','be','bf','bg',
@ -43,7 +50,7 @@ allowed_tags = ('b','blockquote','br','code','del','em','h1','h2','h3','h4',
'tbody','th','thead','td','tr','ul','a','span','ruby','rp','rt',
'spoiler',)
if app.config['MULTIMEDIA_EMBEDDING_ENABLED']:
if MULTIMEDIA_EMBEDDING_ENABLED:
allowed_tags += ('img', 'lite-youtube', 'video', 'source',)
@ -119,11 +126,9 @@ def render_emoji(html, regexp, edit, marseys_used=set(), b=False):
emoji = i.group(1).lower()
attrs = ''
if b: attrs += ' b'
if not edit and len(emojis) <= 20 and random() < 0.0025 and ('marsey' in emoji or emoji in marseys_const2): attrs += ' g'
old = emoji
emoji = emoji.replace('!','').replace('#','')
if emoji == 'marseyrandom': emoji = choice(marseys_const2)
emoji_partial_pat = '<img loading="lazy" alt=":{0}:" src="{1}"{2}>'
emoji_partial = '<img loading="lazy" data-bs-toggle="tooltip" alt=":{0}:" title=":{0}:" src="{1}"{2}>'
@ -182,7 +187,7 @@ def sanitize(sanitized, alert=False, comment=False, edit=False):
# double newlines, eg. hello\nworld becomes hello\n\nworld, which later becomes <p>hello</p><p>world</p>
sanitized = linefeeds_regex.sub(r'\1\n\n\2', sanitized)
if app.config['MULTIMEDIA_EMBEDDING_ENABLED']:
if MULTIMEDIA_EMBEDDING_ENABLED:
# turn eg. https://wikipedia.org/someimage.jpg into ![](https://wikipedia.org/someimage.jpg)
sanitized = image_regex.sub(r'\1![](\2)\4', sanitized)
@ -220,8 +225,8 @@ def sanitize(sanitized, alert=False, comment=False, edit=False):
names = set(m.group(2) for m in matches)
users = get_users(names,graceful=True)
if len(users) > app.config['MENTION_LIMIT']:
abort(400, f'Mentioned {len(users)} users but limit is {app.config["MENTION_LIMIT"]}')
if len(users) > MENTION_LIMIT:
abort(400, f'Mentioned {len(users)} users but limit is {MENTION_LIMIT}')
for u in users:
if not u: continue
@ -232,7 +237,7 @@ def sanitize(sanitized, alert=False, comment=False, edit=False):
soup = BeautifulSoup(sanitized, 'lxml')
if app.config['MULTIMEDIA_EMBEDDING_ENABLED']:
if MULTIMEDIA_EMBEDDING_ENABLED:
for tag in soup.find_all("img"):
if tag.get("src") and not tag["src"].startswith('/pp/'):
tag["loading"] = "lazy"
@ -281,13 +286,13 @@ def sanitize(sanitized, alert=False, comment=False, edit=False):
if "https://youtube.com/watch?v=" in sanitized: sanitized = sanitized.replace("?t=", "&t=")
if app.config['MULTIMEDIA_EMBEDDING_ENABLED']:
if MULTIMEDIA_EMBEDDING_ENABLED:
captured = []
for i in youtube_regex.finditer(sanitized):
if i.group(0) in captured: continue
captured.append(i.group(0))
params = parse_qs(urlparse(i.group(2).replace('&amp;','&')).query)
params = urllib.parse.parse_qs(urllib.parse.urlparse(i.group(2).replace('&amp;','&')).query)
t = params.get('t', params.get('start', [0]))[0]
if isinstance(t, str): t = t.replace('s','')
@ -297,7 +302,7 @@ def sanitize(sanitized, alert=False, comment=False, edit=False):
sanitized = sanitized.replace(i.group(0), htmlsource)
if app.config['MULTIMEDIA_EMBEDDING_ENABLED']:
if MULTIMEDIA_EMBEDDING_ENABLED:
sanitized = video_sub_regex.sub(r'\1<video controls preload="none"><source src="\2"></video>', sanitized)
if comment:
@ -318,8 +323,6 @@ def sanitize(sanitized, alert=False, comment=False, edit=False):
strip=True,
).clean(sanitized)
soup = BeautifulSoup(sanitized, 'lxml')
links = soup.find_all("a")
@ -331,7 +334,7 @@ def sanitize(sanitized, alert=False, comment=False, edit=False):
href = link.get("href")
if not href: continue
url = urlparse(href)
url = urllib.parse.urlparse(href)
domain = url.netloc
url_path = url.path
domain_list.add(domain+url_path)

8
files/helpers/security.py
View file

@ -1,12 +1,11 @@
from werkzeug.security import *
from os import environ
from files.helpers.config.environment import SECRET_KEY
def generate_hash(string):
msg = bytes(string, "utf-16")
return hmac.new(key=bytes(environ.get("MASTER_KEY"), "utf-16"),
return hmac.new(key=bytes(SECRET_KEY, "utf-16"),
msg=msg,
digestmod='md5'
).hexdigest()
@ -18,6 +17,5 @@ def validate_hash(string, hashstr):
def hash_password(password):
return generate_password_hash(
password, method='pbkdf2:sha512', salt_length=8)

18
files/helpers/services.py
View file

@ -2,15 +2,17 @@ import sys
import gevent
from pusher_push_notifications import PushNotifications
from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import Session
from files.classes.leaderboard import (LeaderboardMeta, ReceivedDownvotesLeaderboard,
GivenUpvotesLeaderboard)
from files.__main__ import db_session, service
from files.classes.leaderboard import (GivenUpvotesLeaderboard,
LeaderboardMeta,
ReceivedDownvotesLeaderboard)
from files.helpers.assetcache import assetcache_path
from files.helpers.const import PUSHER_ID, PUSHER_KEY, SITE_FULL, SITE_ID
from files.__main__ import app, db_session
from files.helpers.config.environment import (ENABLE_SERVICES, PUSHER_ID,
PUSHER_KEY, SITE_FULL, SITE_ID)
if PUSHER_ID != 'blahblahblah':
if service.enable_services and ENABLE_SERVICES and PUSHER_ID != 'blahblahblah':
beams_client = PushNotifications(instance_id=PUSHER_ID, secret_key=PUSHER_KEY)
else:
beams_client = None
@ -47,7 +49,7 @@ _lb_given_upvotes_meta = LeaderboardMeta("Upvotes", "given upvotes", "given-upvo
def leaderboard_thread():
global lb_downvotes_received, lb_upvotes_given
db:scoped_session = db_session() # type: ignore
db: Session = db_session()
lb_downvotes_received = ReceivedDownvotesLeaderboard(_lb_received_downvotes_meta, db)
lb_upvotes_given = GivenUpvotesLeaderboard(_lb_given_upvotes_meta, db)
@ -55,5 +57,5 @@ def leaderboard_thread():
db.close()
sys.stdout.flush()
if app.config["ENABLE_SERVICES"]:
if service.enable_services and ENABLE_SERVICES:
gevent.spawn(leaderboard_thread())

4
files/helpers/strings.py
View file

@ -7,9 +7,11 @@ def sql_ilike_clean(my_str):
return my_str.replace(r'\\', '').replace('_', r'\_').replace('%', '').strip()
# this will also just return a bool verbatim
def bool_from_string(input: typing.Union[str, bool]) -> bool:
def bool_from_string(input: typing.Union[str, int, bool]) -> bool:
if isinstance(input, bool):
return input
elif isinstance(input, int):
return bool(input)
if input.lower() in ("yes", "true", "t", "on", "1"):
return True
if input.lower() in ("no", "false", "f", "off", "0"):

57
files/helpers/time.py
View file

@ -1,20 +1,58 @@
import calendar
import time
from datetime import datetime
from datetime import datetime, timedelta
from typing import Final, Union
DATE_FORMAT: Final[str] = '%Y %B %d'
DATETIME_FORMAT: Final[str] = '%Y %B %d %H:%M:%S UTC'
AgeFormattable = Union[int, timedelta]
TimestampFormattable = Union[int, float, datetime, time.struct_time]
def format_datetime(timestamp:TimestampFormattable) -> str:
def format_datetime(timestamp: TimestampFormattable | None) -> str:
return _format_timestamp(timestamp, DATETIME_FORMAT)
def format_date(timestamp:TimestampFormattable) -> str:
def format_date(timestamp: TimestampFormattable | None) -> str:
return _format_timestamp(timestamp, DATE_FORMAT)
def _format_timestamp(timestamp:TimestampFormattable, format:str) -> str:
if isinstance(timestamp, datetime):
def format_age(timestamp: TimestampFormattable | None) -> str:
if timestamp is None:
return ""
timestamp = _make_timestamp(timestamp)
age:int = int(time.time()) - timestamp
if age < 60: return "just now"
if age < 3600:
minutes = int(age / 60)
return f"{minutes}m ago"
if age < 86400:
hours = int(age / 3600)
return f"{hours}hr ago"
if age < 2678400:
days = int(age / 86400)
return f"{days}d ago"
now = time.gmtime()
ctd = time.gmtime(timestamp)
months = now.tm_mon - ctd.tm_mon + 12 * (now.tm_year - ctd.tm_year)
if now.tm_mday < ctd.tm_mday:
months -= 1
if months < 12:
return f"{months}mo ago"
else:
years = int(months / 12)
return f"{years}yr ago"
def _format_timestamp(timestamp: TimestampFormattable | None, format: str) -> str:
if timestamp is None:
return ""
elif isinstance(timestamp, datetime):
return timestamp.strftime(format)
elif isinstance(timestamp, (int, float)):
timestamp = time.gmtime(timestamp)
@ -22,3 +60,12 @@ def _format_timestamp(timestamp:TimestampFormattable, format:str) -> str:
raise TypeError("Invalid argument type (must be one of int, float, "
"datettime, or struct_time)")
return time.strftime(format, timestamp)
def _make_timestamp(timestamp: TimestampFormattable) -> int:
if isinstance(timestamp, (int, float)):
return int(timestamp)
if isinstance(timestamp, datetime):
return int(timestamp.timestamp())
if isinstance(timestamp, time.struct_time):
return calendar.timegm(timestamp)

192
files/helpers/validators.py Normal file
View file

@ -0,0 +1,192 @@
import shutil
import time
import urllib.parse
from dataclasses import dataclass
from typing import Optional
from flask import Request, abort, request
from werkzeug.datastructures import FileStorage
import files.helpers.embeds as embeds
import files.helpers.sanitize as sanitize
from files.helpers.config.environment import SITE_FULL, YOUTUBE_KEY
from files.helpers.config.const import (SUBMISSION_BODY_LENGTH_MAXIMUM,
SUBMISSION_TITLE_LENGTH_MAXIMUM,
SUBMISSION_URL_LENGTH_MAXIMUM)
from files.helpers.content import canonicalize_url2
from files.helpers.media import process_image
def guarded_value(val:str, min_len:int, max_len:int) -> str:
'''
Get request value `val` and ensure it is within length constraints
Requires a request context and either aborts early or returns a good value
'''
raw = request.values.get(val, '').strip()
raw = raw.replace('\u200e', '')
if len(raw) < min_len: abort(400, f"Minimum length for {val} is {min_len}")
if len(raw) > max_len: abort(400, f"Maximum length for {val} is {max_len}")
# TODO: it may make sense to do more sanitisation here
return raw
def int_ranged(val:str, min:int, max:int) -> int:
raw:Optional[int] = request.values.get(val, default=None, type=int)
if raw is None or raw < min or raw > max:
abort(400,
f"Invalid input ('{val}' must be an integer and be between {min} and {max})")
return raw
@dataclass(frozen=True, kw_only=True, slots=True)
class ValidatedSubmissionLike:
title: str
title_html: str
body: str
body_raw: Optional[str]
body_html: str
url: Optional[str]
thumburl: Optional[str]
@property
def embed_slow(self) -> Optional[str]:
url:Optional[str] = self.url
url_canonical: Optional[urllib.parse.ParseResult] = self.url_canonical
if not url or not url_canonical: return None
embed:Optional[str] = None
domain:str = url_canonical.netloc
if domain == "twitter.com":
embed = embeds.twitter(url)
if url.startswith('https://youtube.com/watch?v=') and YOUTUBE_KEY:
embed = embeds.youtube(url)
if SITE_FULL in domain and "/post/" in url and "context" not in url:
id = url.split("/post/")[1]
if "/" in id: id = id.split("/")[0]
embed = str(int(id))
return embed if embed and len(embed) <= 1500 else None
@property
def repost_search_url(self) -> Optional[str]:
search_url = self.url_canonical_str
if not search_url: return None
if search_url.endswith('/'):
search_url = search_url[:-1]
return search_url
@property
def url_canonical(self) -> Optional[urllib.parse.ParseResult]:
if not self.url: return None
return canonicalize_url2(self.url, httpsify=True)
@property
def url_canonical_str(self) -> Optional[str]:
url_canonical:Optional[urllib.parse.ParseResult] = self.url_canonical
if not url_canonical: return None
return url_canonical.geturl()
@classmethod
def from_flask_request(cls,
request:Request,
*,
allow_embedding:bool,
allow_media_url_upload:bool=True,
embed_url_file_key:str="file2",
edit:bool=False) -> "ValidatedSubmissionLike":
'''
Creates the basic structure for a submission and validating it. The
normal submission API has a lot of duplicate code and while this is not
a pretty solution, this essentially forces all submission-likes through
a central interface.
:param request: The Flask Request object.
:param allow_embedding: Whether to allow embedding. This should usually
be the value from the environment.
:param allow_media_url_upload: Whether to allow media URL upload. This
should generally be `True` for submission submitting if file uploads
are allowed and `False` in other contexts (such as editing)
:param embed_url_file_key: The key to use for inline file uploads.
:param edit: The value of `edit` to pass to `sanitize`
'''
def _process_media(file:Optional[FileStorage]) -> tuple[bool, Optional[str], Optional[str]]:
if request.headers.get("cf-ipcountry") == "T1": # forbid Tor uploads
return False, None, None
elif not file:
# We actually care about falseyness, not just `is not None` because
# no attachment is <FileStorage: '' ('application/octet-stream')>
# (at least from Firefox 111).
return False, None, None
elif not file.content_type.startswith('image/'):
abort(415, "Image files only")
name = f'/images/{time.time()}'.replace('.','') + '.webp'
file.save(name)
url:Optional[str] = process_image(name)
if not url: return False, None, None
name2 = name.replace('.webp', 'r.webp')
shutil.copyfile(name, name2)
thumburl:Optional[str] = process_image(name2, resize=100)
return True, url, thumburl
def _process_media2(body:str, file2:Optional[list[FileStorage]]) -> tuple[bool, str]:
if request.headers.get("cf-ipcountry") == "T1": # forbid Tor uploads
return False, body
elif not file2: # empty list or None
return False, body
file2 = file2[:4]
if not all(file for file in file2):
# Falseyness check to handle <'' ('application/octet-stream')>
return False, body
for file in file2:
if not file.content_type.startswith('image/'):
abort(415, "Image files only")
name = f'/images/{time.time()}'.replace('.','') + '.webp'
file.save(name)
image = process_image(name)
if allow_embedding:
body += f"\n\n![]({image})"
else:
body += f'\n\n<a href="{image}">{image}</a>'
return True, body
title = guarded_value("title", 1, SUBMISSION_TITLE_LENGTH_MAXIMUM)
title = sanitize.sanitize_raw(title, allow_newlines=False, length_limit=SUBMISSION_TITLE_LENGTH_MAXIMUM)
url = guarded_value("url", 0, SUBMISSION_URL_LENGTH_MAXIMUM)
body_raw = guarded_value("body", 0, SUBMISSION_BODY_LENGTH_MAXIMUM)
body_raw = sanitize.sanitize_raw(body_raw, allow_newlines=True, length_limit=SUBMISSION_BODY_LENGTH_MAXIMUM)
if not url and allow_media_url_upload:
has_file, url, thumburl = _process_media(request.files.get("file"))
else:
has_file = False
thumburl = None
has_file2, body = _process_media2(body_raw, request.files.getlist(embed_url_file_key))
if not body_raw and not url and not has_file and not has_file2:
raise ValueError("Please enter a URL or some text")
title_html = sanitize.filter_emojis_only(title, graceful=True)
if len(title_html) > 1500:
raise ValueError("Rendered title is too big!")
return ValidatedSubmissionLike(
title=title,
title_html=sanitize.filter_emojis_only(title, graceful=True),
body=body,
body_raw=body_raw,
body_html=sanitize.sanitize(body, edit=edit),
url=url,
thumburl=thumburl,
)

45
files/helpers/wrappers.py
View file

@ -1,11 +1,16 @@
from .get import *
from .alerts import *
from files.helpers.const import *
from files.__main__ import db_session
from random import randint
import user_agents
import functools
import time
import user_agents
from files.helpers.alerts import *
from files.helpers.config.const import *
from files.helpers.config.environment import SITE
from files.helpers.get import *
from files.routes.importstar import *
from files.__main__ import app, cache, db_session
def get_logged_in_user():
if hasattr(g, 'v'):
return g.v
@ -80,68 +85,52 @@ def get_logged_in_user():
g.v = v
return v
def check_ban_evade(v):
if v and not v.patron and v.admin_level < 2 and v.ban_evade and not v.unban_utc:
v.shadowbanned = "AutoJanny"
g.db.add(v)
g.db.commit()
def auth_desired(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
v = get_logged_in_user()
check_ban_evade(v)
return make_response(f(*args, v=v, **kwargs))
wrapper.__name__ = f.__name__
return wrapper
def auth_required(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
v = get_logged_in_user()
if not v: abort(401)
check_ban_evade(v)
return make_response(f(*args, v=v, **kwargs))
wrapper.__name__ = f.__name__
return wrapper
def is_not_permabanned(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
v = get_logged_in_user()
if not v: abort(401)
check_ban_evade(v)
if v.is_suspended_permanently:
abort(403, "You are permanently banned")
return make_response(f(*args, v=v, **kwargs))
wrapper.__name__ = f.__name__
return wrapper
def admin_level_required(x):
def wrapper_maker(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
v = get_logged_in_user()
if not v: abort(401)
if v.admin_level < x: abort(403)
return make_response(f(*args, v=v, **kwargs))
wrapper.__name__ = f.__name__
return wrapper
return wrapper_maker

25
files/mail/__init__.py
View file

@ -1,30 +1,28 @@
from os import environ
import time
from flask import *
from urllib.parse import quote
from files.helpers.security import *
from files.helpers.wrappers import *
from files.helpers.const import *
from files.classes import *
from files.__main__ import app, mail, limiter
from flask_mail import Message
SITE_ID = environ.get("SITE_ID").strip()
SITE_TITLE = environ.get("SITE_TITLE").strip()
from files.__main__ import app, limiter, mail
from files.classes.badges import Badge
from files.classes.user import User
from files.helpers.config.const import *
from files.helpers.config.environment import SERVER_NAME, SITE_ID, SITE_TITLE
from files.helpers.security import *
from files.helpers.wrappers import *
from files.routes.importstar import *
def send_mail(to_address, subject, html):
msg = Message(html=html, subject=subject, sender=f"{SITE_ID}@{SITE}", recipients=[to_address])
mail.send(msg)
def send_verification_email(user, email=None):
if not email:
email = user.email
url = f"https://{app.config['SERVER_NAME']}/activate"
url = f"https://{SERVER_NAME}/activate"
now = int(time.time())
token = generate_hash(f"{email}+{user.id}+{now}")
@ -44,16 +42,13 @@ def send_verification_email(user, email=None):
@limiter.limit("1/second;30/minute;200/hour;1000/day")
@auth_required
def api_verify_email(v):
send_verification_email(v)
return {"message": "Email has been sent (ETA ~5 minutes)"}
@app.get("/activate")
@auth_required
def activate(v):
email = request.values.get("email", "").strip().lower()
if not email_regex.fullmatch(email):

1
files/routes/__init__.py
View file

@ -1,4 +1,5 @@
from files.__main__ import app
from files.helpers.config.const import FEATURES
from .admin import *
from .comments import *

1
files/routes/admin/__init__.py
View file

@ -1,2 +1,3 @@
from .admin import *
from .performance import *
from .tasks import *

35
files/routes/admin/admin.py
View file

@ -1,19 +1,22 @@
import json
import time
from datetime import datetime
from files.helpers.wrappers import *
import requests
from files.classes import *
from files.helpers.alerts import *
from files.helpers.sanitize import *
from files.helpers.security import *
from files.helpers.caching import invalidate_cache
from files.helpers.comments import comment_on_publish, comment_on_unpublish
from files.helpers.config.const import *
from files.helpers.config.environment import CF_HEADERS, CF_ZONE
from files.helpers.get import *
from files.helpers.media import *
from files.helpers.const import *
from files.classes import *
from flask import *
from files.helpers.sanitize import *
from files.helpers.security import *
from files.helpers.wrappers import *
from files.__main__ import app, cache, limiter
from ..front import frontlist
from files.helpers.comments import comment_on_publish, comment_on_unpublish
from datetime import datetime
import requests
from files.routes.importstar import *
month = datetime.now().strftime('%B')
@ -830,7 +833,7 @@ def shadowban(user_id, v):
)
g.db.add(ma)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
body = f"@{v.username} has shadowbanned @{user.username}"
@ -878,7 +881,7 @@ def unshadowban(user_id, v):
)
g.db.add(ma)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()
return {"message": "User unshadowbanned!"}
@ -1108,7 +1111,7 @@ def ban_post(post_id, v):
)
g.db.add(ma)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
v.coins += 1
g.db.add(v)
@ -1144,7 +1147,7 @@ def unban_post(post_id, v):
g.db.add(post)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
v.coins -= 1
g.db.add(v)
@ -1213,7 +1216,7 @@ def sticky_post(post_id, v):
if v.id != post.author_id:
send_repeatable_notification(post.author_id, f"@{v.username} has pinned your [post](/post/{post_id})!")
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()
return {"message": "Post pinned!"}
@ -1239,7 +1242,7 @@ def unsticky_post(post_id, v):
if v.id != post.author_id:
send_repeatable_notification(post.author_id, f"@{v.username} has unpinned your [post](/post/{post_id})!")
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()
return {"message": "Post unpinned!"}

2
files/routes/admin/performance.py
View file

@ -6,7 +6,7 @@ from typing import Final
import psutil
from flask import abort, render_template, request
from files.helpers.const import PERMS
from files.helpers.config.const import PERMS
from files.helpers.time import format_datetime
from files.helpers.wrappers import admin_level_required
from files.__main__ import app

179
files/routes/admin/tasks.py Normal file
View file

@ -0,0 +1,179 @@
from datetime import time
from typing import Optional
from flask import abort, g, redirect, render_template, request
import files.helpers.validators as validators
from files.__main__ import app
from files.classes.cron.submission import ScheduledSubmissionTask
from files.classes.cron.tasks import (DayOfWeek, RepeatableTask,
RepeatableTaskRun, ScheduledTaskType)
from files.classes.user import User
from files.helpers.config.const import PERMS, SUBMISSION_FLAIR_LENGTH_MAXIMUM
from files.helpers.config.environment import MULTIMEDIA_EMBEDDING_ENABLED
from files.helpers.wrappers import admin_level_required
def _modify_task_schedule(pid:int):
task: Optional[RepeatableTask] = g.db.get(RepeatableTask, pid)
if not task: abort(404)
# rebuild the schedule
task.enabled = _get_request_bool('enabled')
task.frequency_day_flags = _get_request_dayofweek()
hour:int = validators.int_ranged('hour', 0, 23)
minute:int = validators.int_ranged('minute', 0, 59)
second:int = 0 # TODO: seconds?
time_of_day_utc:time = time(hour, minute, second)
task.time_of_day_utc = time_of_day_utc
g.db.commit()
@app.get('/tasks/')
@admin_level_required(PERMS['SCHEDULER'])
def tasks_get(v:User):
tasks:list[RepeatableTask] = \
g.db.query(RepeatableTask).all()
return render_template("admin/tasks/tasks.html", v=v, listing=tasks)
@app.get('/tasks/<int:task_id>/')
@admin_level_required(PERMS['SCHEDULER'])
def tasks_get_task(v:User, task_id:int):
task:RepeatableTask = g.db.get(RepeatableTask, task_id)
if not task: abort(404)
return render_template("admin/tasks/single_task.html", v=v, task=task)
@app.get('/tasks/<int:task_id>/runs/')
@admin_level_required(PERMS['SCHEDULER'])
def tasks_get_task_redirect(v:User, task_id:int): # pyright: ignore
return redirect(f'/tasks/{task_id}/')
@app.get('/tasks/<int:task_id>/runs/<int:run_id>')
@admin_level_required(PERMS['SCHEDULER'])
def tasks_get_task_run(v:User, task_id:int, run_id:int):
run:RepeatableTaskRun = g.db.get(RepeatableTaskRun, run_id)
if not run: abort(404)
if run.task_id != task_id:
return redirect(f'/tasks/{run.task_id}/runs/{run.id}')
return render_template("admin/tasks/single_run.html", v=v, run=run)
@app.post('/tasks/<int:task_id>/schedule')
@admin_level_required(PERMS['SCHEDULER'])
def task_schedule_post(v:User, task_id:int): # pyright: ignore
_modify_task_schedule(task_id)
return redirect(f'/tasks/{task_id}')
@app.get('/tasks/scheduled_posts/')
@admin_level_required(PERMS['SCHEDULER_POSTS'])
def tasks_scheduled_posts_get(v:User):
submissions:list[ScheduledSubmissionTask] = \
g.db.query(ScheduledSubmissionTask).all()
return render_template("admin/tasks/scheduled_posts.html", v=v, listing=submissions)
def _get_request_bool(name:str) -> bool:
return bool(request.values.get(name, default=False, type=bool))
def _get_request_dayofweek() -> DayOfWeek:
days:DayOfWeek = DayOfWeek.NONE
for day in DayOfWeek.all_days:
name:str = day.name.lower()
if _get_request_bool(f'schedule_day_{name}'): days |= day
return days
@app.post('/tasks/scheduled_posts/')
@admin_level_required(PERMS['SCHEDULER_POSTS'])
def tasks_scheduled_posts_post(v:User):
validated_post:validators.ValidatedSubmissionLike = \
validators.ValidatedSubmissionLike.from_flask_request(request,
allow_embedding=MULTIMEDIA_EMBEDDING_ENABLED,
)
# first build the template
flair:str = validators.guarded_value("flair", min_len=0, max_len=SUBMISSION_FLAIR_LENGTH_MAXIMUM)
# and then build the schedule
enabled:bool = _get_request_bool('enabled')
frequency_day:DayOfWeek = _get_request_dayofweek()
hour:int = validators.int_ranged('hour', 0, 23)
minute:int = validators.int_ranged('minute', 0, 59)
second:int = 0 # TODO: seconds?
time_of_day_utc:time = time(hour, minute, second)
# and then build the scheduled task
task:ScheduledSubmissionTask = ScheduledSubmissionTask(
author_id=v.id,
author_id_submission=v.id, # TODO: allow customization
enabled=enabled,
ghost=_get_request_bool("ghost"),
private=_get_request_bool("private"),
over_18=_get_request_bool("over_18"),
is_bot=False, # TODO: do we need this?
title=validated_post.title,
url=validated_post.url,
body=validated_post.body,
body_html=validated_post.body_html,
flair=flair,
embed_url=validated_post.embed_slow,
frequency_day=int(frequency_day),
time_of_day_utc=time_of_day_utc,
type_id=int(ScheduledTaskType.SCHEDULED_SUBMISSION),
)
g.db.add(task)
g.db.commit()
return redirect(f'/tasks/scheduled_posts/{task.id}')
@app.get('/tasks/scheduled_posts/<int:pid>')
@admin_level_required(PERMS['SCHEDULER_POSTS'])
def tasks_scheduled_post_get(v:User, pid:int):
submission: Optional[ScheduledSubmissionTask] = \
g.db.get(ScheduledSubmissionTask, pid)
if not submission: abort(404)
return render_template("admin/tasks/scheduled_post.html", v=v, p=submission)
@app.post('/tasks/scheduled_posts/<int:pid>/content')
@admin_level_required(PERMS['SCHEDULER_POSTS'])
def task_scheduled_post_content_post(v:User, pid:int): # pyright: ignore
submission: Optional[ScheduledSubmissionTask] = \
g.db.get(ScheduledSubmissionTask, pid)
if not submission: abort(404)
if not v.can_edit(submission): abort(403)
validated_post:validators.ValidatedSubmissionLike = \
validators.ValidatedSubmissionLike.from_flask_request(request,
allow_embedding=MULTIMEDIA_EMBEDDING_ENABLED,
)
edited:bool = False
if submission.body != validated_post.body:
submission.body = validated_post.body
submission.body_html = validated_post.body_html
edited = True
if submission.title != validated_post.title:
submission.title = validated_post.title
edited = True
if not edited:
abort(400, "Title or body must be edited")
g.db.commit()
return redirect(f'/tasks/scheduled_posts/{pid}')
@app.post('/tasks/scheduled_posts/<int:task_id>/schedule')
@admin_level_required(PERMS['SCHEDULER'])
def task_scheduled_post_post(v:User, task_id:int): # pyright: ignore
# permission being SCHEDULER is intentional as SCHEDULER_POSTS is for
# creating or editing post content
_modify_task_schedule(task_id)
return redirect(f'/tasks/scheduled_posts/{task_id}')

51
files/routes/allroutes.py Normal file
View file

@ -0,0 +1,51 @@
import json
import sys
import time
from flask import abort, g, request
from files.__main__ import app, db_session, limiter
@app.before_request
def before_request():
with open('site_settings.json', 'r') as f:
app.config['SETTINGS'] = json.load(f)
if request.host != app.config["SERVER_NAME"]:
return {"error": "Unauthorized host provided."}, 403
if not app.config['SETTINGS']['Bots'] and request.headers.get("Authorization"):
abort(403, "Bots are currently not allowed")
g.agent = request.headers.get("User-Agent")
if not g.agent:
return 'Please use a "User-Agent" header!', 403
ua = g.agent.lower()
g.debug = app.debug
g.webview = ('; wv) ' in ua)
g.inferior_browser = (
'iphone' in ua or
'ipad' in ua or
'ipod' in ua or
'mac os' in ua or
' firefox/' in ua)
g.timestamp = int(time.time())
limiter.check()
g.db = db_session()
@app.teardown_appcontext
def teardown_request(error):
if hasattr(g, 'db') and g.db:
g.db.close()
sys.stdout.flush()
@app.after_request
def after_request(response):
response.headers.add("Strict-Transport-Security", "max-age=31536000")
response.headers.add("X-Frame-Options", "deny")
return response

7
files/routes/awards.py
View file

@ -2,9 +2,8 @@ from files.__main__ import app, limiter
from files.helpers.wrappers import *
from files.helpers.alerts import *
from files.helpers.get import *
from files.helpers.const import *
from files.helpers.config.const import *
from files.classes.award import *
from .front import frontlist
from flask import g, request
from files.helpers.sanitize import filter_emojis_only
from copy import deepcopy
@ -24,7 +23,7 @@ def shop(v):
for val in AWARDS.values():
val["baseprice"] = int(val["price"])
val["price"] = int(val["price"] * v.discount)
val["price"] = val["baseprice"]
sales = g.db.query(func.sum(User.coins_spent)).scalar()
return render_template("shop.html", awards=list(AWARDS.values()), v=v, sales=sales)
@ -46,7 +45,7 @@ def buy(v, award):
if award not in AWARDS: abort(400)
og_price = AWARDS[award]["price"]
price = int(og_price * v.discount)
price = int(og_price)
if request.values.get("mb"):
if v.procoins < price: abort(400, "Not enough marseybux.")

3
files/routes/chat.py
View file

@ -1,7 +1,8 @@
import time
from files.helpers.config.environment import SITE, SITE_FULL
from files.helpers.wrappers import auth_required
from files.helpers.sanitize import sanitize
from files.helpers.const import *
from files.helpers.config.const import *
from datetime import datetime
from flask_socketio import SocketIO, emit
from files.__main__ import app, limiter, cache

30
files/routes/comments.py
View file

@ -1,16 +1,12 @@
from files.helpers.wrappers import *
from files.helpers.alerts import *
from files.helpers.media import process_image
from files.helpers.const import *
from files.helpers.comments import comment_on_publish
from files.classes import *
from flask import *
from files.__main__ import app, limiter
from files.helpers.sanitize import filter_emojis_only
import requests
from shutil import copyfile
from json import loads
from collections import Counter
from files.classes import *
from files.helpers.alerts import *
from files.helpers.comments import comment_on_publish
from files.helpers.config.const import *
from files.helpers.media import process_image
from files.helpers.wrappers import *
from files.routes.importstar import *
@app.get("/comment/<cid>")
@app.get("/post/<pid>/<anything>/<cid>")
@ -183,9 +179,9 @@ def api_comment(v):
).all()
threshold = app.config["COMMENT_SPAM_COUNT_THRESHOLD"]
if v.age >= (60 * 60 * 24 * 7):
if v.age_seconds >= (60 * 60 * 24 * 7):
threshold *= 3
elif v.age >= (60 * 60 * 24):
elif v.age_seconds >= (60 * 60 * 24):
threshold *= 2
if len(similar_comments) > threshold:
@ -280,11 +276,11 @@ def edit_comment(cid, v):
).all()
threshold = app.config["SPAM_SIMILAR_COUNT_THRESHOLD"]
if v.age >= (60 * 60 * 24 * 30):
if v.age_seconds >= (60 * 60 * 24 * 30):
threshold *= 4
elif v.age >= (60 * 60 * 24 * 7):
elif v.age_seconds >= (60 * 60 * 24 * 7):
threshold *= 3
elif v.age >= (60 * 60 * 24):
elif v.age_seconds >= (60 * 60 * 24):
threshold *= 2
if len(similar_comments) > threshold:

2
files/routes/dev.py
View file

@ -1,7 +1,7 @@
from secrets import token_hex
from flask import session, redirect, request
from files.helpers.const import PERMS
from files.helpers.config.const import PERMS
from files.helpers.get import get_user
from files.helpers.wrappers import admin_level_required
from files.__main__ import app

6
files/routes/errors.py
View file

@ -4,8 +4,11 @@ from urllib.parse import quote, urlencode
from flask import g, redirect, render_template, request, session
from files.helpers.const import ERROR_MESSAGES, SITE_FULL, WERKZEUG_ERROR_DESCRIPTIONS
from files.__main__ import app
from files.helpers.config.const import (ERROR_MESSAGES,
WERKZEUG_ERROR_DESCRIPTIONS)
from files.helpers.config.environment import SITE_FULL
@app.errorhandler(400)
@app.errorhandler(401)
@ -14,6 +17,7 @@ from files.__main__ import app
@app.errorhandler(405)
@app.errorhandler(409)
@app.errorhandler(413)
@app.errorhandler(415)
@app.errorhandler(422)
@app.errorhandler(429)
def error(e):

17
files/routes/feeds.py
View file

@ -1,13 +1,14 @@
import html
from .front import frontlist
from datetime import datetime
from files.helpers.get import *
from yattag import Doc
from files.helpers.const import *
from files.helpers.wrappers import *
from files.helpers.jinja2 import *
from yattag import Doc
import files.helpers.listing as listing
from files.__main__ import app
from files.helpers.config.const import *
from files.helpers.get import *
from files.helpers.jinja2 import *
from files.helpers.wrappers import *
@app.get('/rss')
@app.get('/feed')
@ -16,7 +17,7 @@ def feeds_front(sort='new', t='all'):
try: page = max(int(request.values.get("page", 1)), 1)
except: page = 1
ids, next_exists = frontlist(
ids, next_exists = listing.frontlist(
sort=sort,
page=page,
t=t,

125
files/routes/front.py
View file

@ -1,15 +1,14 @@
from sqlalchemy.orm import Query
from files.helpers.wrappers import *
from files.helpers.get import *
from files.helpers.strings import sql_ilike_clean
from files.__main__ import app, cache, limiter
import files.helpers.listing as listing
from files.__main__ import app, limiter
from files.classes.submission import Submission
from files.helpers.comments import comment_filter_moderated
from files.helpers.contentsorting import \
apply_time_filter, sort_objects, sort_comment_results
defaulttimefilter = environ.get("DEFAULT_TIME_FILTER", "all").strip()
from files.helpers.contentsorting import (apply_time_filter,
sort_comment_results, sort_objects)
from files.helpers.config.environment import DEFAULT_TIME_FILTER
from files.helpers.get import *
from files.helpers.wrappers import *
@app.post("/clear")
@auth_required
@ -198,7 +197,7 @@ def front_all(v, sub=None, subdomain=None):
try: lt=int(request.values.get("before", 0))
except: lt=0
ids, next_exists = frontlist(sort=sort,
ids, next_exists = listing.frontlist(sort=sort,
page=page,
t=t,
v=v,
@ -233,90 +232,6 @@ def front_all(v, sub=None, subdomain=None):
return render_template("home.html", v=v, listing=posts, next_exists=next_exists, sort=sort, t=t, page=page, ccmode=ccmode, sub=sub, home=True)
@cache.memoize(timeout=86400)
def frontlist(v=None, sort='new', page=1, t="all", ids_only=True, ccmode="false", filter_words='', gt=0, lt=0, sub=None, site=None):
posts = g.db.query(Submission)
if v and v.hidevotedon:
voted = [x[0] for x in g.db.query(Vote.submission_id).filter_by(user_id=v.id).all()]
posts = posts.filter(Submission.id.notin_(voted))
if not v or v.admin_level < 2:
filter_clause = (Submission.filter_state != 'filtered') & (Submission.filter_state != 'removed')
if v:
filter_clause = filter_clause | (Submission.author_id == v.id)
posts = posts.filter(filter_clause)
if sub: posts = posts.filter_by(sub=sub.name)
elif v: posts = posts.filter(or_(Submission.sub == None, Submission.sub.notin_(v.all_blocks)))
if gt: posts = posts.filter(Submission.created_utc > gt)
if lt: posts = posts.filter(Submission.created_utc < lt)
if not gt and not lt:
posts = apply_time_filter(posts, t, Submission)
if (ccmode == "true"):
posts = posts.filter(Submission.club == True)
posts = posts.filter_by(is_banned=False, private=False, deleted_utc = 0)
if ccmode == "false" and not gt and not lt:
posts = posts.filter_by(stickied=None)
if v and v.admin_level < 2:
posts = posts.filter(Submission.author_id.notin_(v.userblocks))
if not (v and v.changelogsub):
posts=posts.filter(not_(Submission.title.ilike('[changelog]%')))
if v and filter_words:
for word in filter_words:
word = sql_ilike_clean(word).strip()
posts=posts.filter(not_(Submission.title.ilike(f'%{word}%')))
if not (v and v.shadowbanned):
posts = posts.join(User, User.id == Submission.author_id).filter(User.shadowbanned == None)
posts = sort_objects(posts, sort, Submission)
if v: size = v.frontsize or 0
else: size = 25
posts = posts.offset(size * (page - 1)).limit(size+1).all()
next_exists = (len(posts) > size)
posts = posts[:size]
if page == 1 and ccmode == "false" and not gt and not lt:
pins = g.db.query(Submission).filter(Submission.stickied != None, Submission.is_banned == False)
if sub: pins = pins.filter_by(sub=sub.name)
elif v:
pins = pins.filter(or_(Submission.sub == None, Submission.sub.notin_(v.all_blocks)))
if v.admin_level < 2:
pins = pins.filter(Submission.author_id.notin_(v.userblocks))
pins = pins.all()
for pin in pins:
if pin.stickied_utc and int(time.time()) > pin.stickied_utc:
pin.stickied = None
pin.stickied_utc = None
g.db.add(pin)
pins.remove(pin)
posts = pins + posts
if ids_only: posts = [x.id for x in posts]
g.db.commit()
return posts, next_exists
@app.get("/changelog")
@auth_required
def changelog(v):
@ -326,7 +241,7 @@ def changelog(v):
sort=request.values.get("sort", "new")
t=request.values.get('t', "all")
ids = changeloglist(sort=sort,
ids = listing.changeloglist(sort=sort,
page=page,
t=t,
v=v,
@ -342,26 +257,6 @@ def changelog(v):
return render_template("changelog.html", v=v, listing=posts, next_exists=next_exists, sort=sort, t=t, page=page)
@cache.memoize(timeout=86400)
def changeloglist(v=None, sort="new", page=1, t="all", site=None):
posts = g.db.query(Submission.id).filter_by(is_banned=False, private=False,).filter(Submission.deleted_utc == 0)
if v.admin_level < 2:
posts = posts.filter(Submission.author_id.notin_(v.userblocks))
admins = [x[0] for x in g.db.query(User.id).filter(User.admin_level > 0).all()]
posts = posts.filter(Submission.title.ilike('_changelog%'), Submission.author_id.in_(admins))
if t != 'all':
posts = apply_time_filter(posts, t, Submission)
posts = sort_objects(posts, sort, Submission)
posts = posts.offset(25 * (page - 1)).limit(26).all()
return [x[0] for x in posts]
@app.get("/random_post")
def random_post():
p = g.db.query(Submission.id).filter(Submission.deleted_utc == 0, Submission.is_banned == False, Submission.private == False).order_by(func.random()).first()
@ -387,7 +282,7 @@ def random_user():
def all_comments(v):
page = max(request.values.get("page", 1, int), 1)
sort = request.values.get("sort", "new")
time_filter = request.values.get("t", defaulttimefilter)
time_filter = request.values.get("t", DEFAULT_TIME_FILTER)
time_gt = request.values.get("after", 0, int)
time_lt = request.values.get("before", 0, int)

17
files/routes/importstar.py Normal file
View file

@ -0,0 +1,17 @@
'''
Module that can be safely imported with the syntax
`from files.routes.importstar import *`. This essentially
contains flask stuff for routes that are used by pretty much
all routes.
This should only be used from the route handlers. Flask imports
are used in pretty much every place, but they shouldn't be used
from the models if at all possible.
Ideally we'd import only what we need but this is just for ease
of development. Feel free to remove.
'''
from flask import (Response, abort, g, jsonify, make_response, redirect,
render_template, request, send_file, send_from_directory,
session)

12
files/routes/login.py
View file

@ -1,7 +1,8 @@
from urllib.parse import urlencode
from files.helpers.config.environment import HCAPTCHA_SECRET, HCAPTCHA_SITEKEY, WELCOME_MSG
from files.mail import *
from files.__main__ import app, limiter
from files.helpers.const import *
from files.helpers.config.const import *
from files.helpers.captcha import validate_captcha
@app.get("/login")
@ -201,7 +202,7 @@ def sign_up_get(v):
formkey_hashstr = str(now) + token + agent
formkey = hmac.new(key=bytes(environ.get("MASTER_KEY"), "utf-16"),
formkey = hmac.new(key=bytes(SECRET_KEY, "utf-16"),
msg=bytes(formkey_hashstr, "utf-16"),
digestmod='md5'
).hexdigest()
@ -212,7 +213,7 @@ def sign_up_get(v):
formkey=formkey,
now=now,
ref_user=ref_user,
hcaptcha=app.config["HCAPTCHA_SITEKEY"],
hcaptcha=HCAPTCHA_SITEKEY,
error=error
)
@ -237,7 +238,7 @@ def sign_up_post(v):
correct_formkey_hashstr = form_timestamp + submitted_token + agent
correct_formkey = hmac.new(key=bytes(environ.get("MASTER_KEY"), "utf-16"),
correct_formkey = hmac.new(key=bytes(SECRET_KEY, "utf-16"),
msg=bytes(correct_formkey_hashstr, "utf-16"),
digestmod='md5'
).hexdigest()
@ -289,8 +290,7 @@ def sign_up_post(v):
if existing_account:
return signup_error("An account with that username already exists.")
if not validate_captcha(app.config.get("HCAPTCHA_SECRET", ""),
app.config.get("HCAPTCHA_SITEKEY", ""),
if not validate_captcha(HCAPTCHA_SECRET, HCAPTCHA_SITEKEY,
request.values.get("h-captcha-response", "")):
return signup_error("Unable to verify CAPTCHA")

25
files/routes/oauth.py
View file

@ -1,12 +1,14 @@
from files.helpers.wrappers import *
from files.helpers.alerts import *
from files.helpers.get import *
from files.helpers.const import *
from files.classes import *
from flask import *
from files.__main__ import app, limiter
import sqlalchemy.exc
from files.__main__ import app, limiter
from files.classes import *
from files.helpers.alerts import *
from files.helpers.config.const import *
from files.helpers.get import *
from files.helpers.wrappers import *
from files.routes.importstar import *
@app.get("/authorize")
@auth_required
def authorize_prompt(v):
@ -208,9 +210,6 @@ def admin_app_reject(v, aid):
@app.get("/admin/app/<aid>")
@admin_level_required(2)
def admin_app_id(v, aid):
aid=aid
oauth = g.db.query(OauthApp).filter_by(id=aid).one_or_none()
pids=oauth.idlist(page=int(request.values.get("page",1)))
@ -230,13 +229,9 @@ def admin_app_id(v, aid):
@app.get("/admin/app/<aid>/comments")
@admin_level_required(2)
def admin_app_id_comments(v, aid):
aid=aid
oauth = g.db.query(OauthApp).filter_by(id=aid).one_or_none()
cids=oauth.comments_idlist(page=int(request.values.get("page",1)),
)
cids=oauth.comments_idlist(page=int(request.values.get("page",1)))
next_exists=len(cids)==101
cids=cids[:100]

545
files/routes/posts.py
View file

@ -1,61 +1,40 @@
import time
import gevent
from files.helpers.wrappers import *
from files.helpers.sanitize import *
from files.helpers.alerts import *
from files.helpers.comments import comment_filter_moderated
from files.helpers.contentsorting import sort_objects
from files.helpers.const import *
from files.helpers.media import process_image
from files.helpers.strings import sql_ilike_clean
from files.classes import *
from flask import *
import urllib.parse
from io import BytesIO
from files.__main__ import app, limiter, cache, db_session
from PIL import Image as PILimage
from .front import frontlist, changeloglist
from urllib.parse import ParseResult, urlunparse, urlparse, quote, unquote
from os import path
import requests
from shutil import copyfile
from sys import stdout
from urllib.parse import ParseResult, urlparse
import gevent
import requests
import werkzeug.wrappers
from PIL import Image as PILimage
from sqlalchemy.orm import Query
import files.helpers.validators as validators
from files.__main__ import app, cache, db_session, limiter
from files.classes import *
from files.helpers.alerts import *
from files.helpers.caching import invalidate_cache
from files.helpers.comments import comment_filter_moderated
from files.helpers.config.const import *
from files.helpers.content import canonicalize_url2
from files.helpers.contentsorting import sort_objects
from files.helpers.media import process_image
from files.helpers.sanitize import *
from files.helpers.strings import sql_ilike_clean
from files.helpers.wrappers import *
from files.routes.importstar import *
snappyquotes = [f':#{x}:' for x in marseys_const2]
if path.exists(f'snappy_{SITE_ID}.txt'):
with open(f'snappy_{SITE_ID}.txt', "r", encoding="utf-8") as f:
snappyquotes += f.read().split("\n{[para]}\n")
discounts = {
69: 0.02,
70: 0.04,
71: 0.06,
72: 0.08,
73: 0.10,
titleheaders = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36"
}
titleheaders = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36"}
MAX_TITLE_LENGTH = 500
MAX_URL_LENGTH = 2048
MAX_BODY_LENGTH = SUBMISSION_BODY_LENGTH_MAXIMUM
def guarded_value(val, min_len, max_len) -> str:
'''
Get request value `val` and ensure it is within length constraints
Requires a request context and either aborts early or returns a good value
'''
raw = request.values.get(val, '').strip()
raw = raw.replace('\u200e', '')
if len(raw) < min_len: abort(400, f"Minimum length for {val} is {min_len}")
if len(raw) > max_len: abort(400, f"Maximum length for {val} is {max_len}")
# TODO: it may make sense to do more sanitisation here
return raw
@app.post("/toggle_club/<pid>")
@auth_required
def toggle_club(pid, v):
@ -84,32 +63,8 @@ def publish(pid, v):
post.created_utc = int(time.time())
g.db.add(post)
if not post.ghost:
notify_users = NOTIFY_USERS(f'{post.title} {post.body}', v)
if notify_users:
cid = notif_comment2(post)
for x in notify_users:
add_notif(cid, x)
if v.followers:
text = f"@{v.username} has made a new post: [{post.title}]({post.shortlink})"
if post.sub: text += f" in <a href='/h/{post.sub}'>/h/{post.sub}"
cid = notif_comment(text, autojanny=True)
for follow in v.followers:
user = get_account(follow.user_id)
if post.club and not user.paid_dues: continue
add_notif(cid, user.id)
post.publish()
g.db.commit()
cache.delete_memoized(frontlist)
cache.delete_memoized(User.userpagelisting)
if v.admin_level > 0 and ("[changelog]" in post.title.lower() or "(changelog)" in post.title.lower()):
cache.delete_memoized(changeloglist)
return redirect(post.permalink)
@app.get("/submit")
@ -294,49 +249,37 @@ def morecomments(v, cid):
return render_template("comments.html", v=v, comments=comments, p=p, render_replies=True, ajax=True)
@app.post("/edit_post/<pid>")
@app.post("/edit_post/<int:pid>")
@limiter.limit("1/second;30/minute;200/hour;1000/day")
@auth_required
def edit_post(pid, v):
p = get_post(pid)
if not v.can_edit(p): abort(403)
if p.author_id != v.id and not (v.admin_level > 1 and v.admin_level > 2): abort(403)
validated_post: validators.ValidatedSubmissionLike = \
validators.ValidatedSubmissionLike.from_flask_request(
request,
allow_embedding=MULTIMEDIA_EMBEDDING_ENABLED,
allow_media_url_upload=False,
embed_url_file_key="file",
edit=True
)
changed:bool=False
title = guarded_value("title", 1, MAX_TITLE_LENGTH)
title = sanitize_raw(title, allow_newlines=False, length_limit=MAX_TITLE_LENGTH)
if validated_post.title != p.title:
p.title = validated_post.title
p.title_html = validated_post.title_html
changed = True
body = guarded_value("body", 0, MAX_BODY_LENGTH)
body = sanitize_raw(body, allow_newlines=True, length_limit=MAX_BODY_LENGTH)
if validated_post.body != p.body:
p.body = validated_post.body
p.body_html = validated_post.body_html
changed = True
if title != p.title:
p.title = title
title_html = filter_emojis_only(title, edit=True)
p.title_html = title_html
if not changed:
abort(400, "You need to change something")
if request.files.get("file") and request.headers.get("cf-ipcountry") != "T1":
files = request.files.getlist('file')[:4]
for file in files:
if file.content_type.startswith('image/'):
name = f'/images/{time.time()}'.replace('.','') + '.webp'
file.save(name)
url = process_image(name)
if app.config['MULTIMEDIA_EMBEDDING_ENABLED']:
body += f"\n\n![]({url})"
else:
body += f'\n\n<a href="{url}">{url}</a>'
else: abort(400, "Image files only")
body_html = sanitize(body, edit=True)
p.body = body
p.body_html = body_html
if not p.private and not p.ghost:
notify_users = NOTIFY_USERS(f'{p.title} {p.body}', v)
if notify_users:
cid = notif_comment2(p)
for x in notify_users:
add_notif(cid, x)
p.publish()
if v.id == p.author_id:
if int(time.time()) - p.created_utc > 60 * 3: p.edited_utc = int(time.time())
@ -353,17 +296,16 @@ def edit_post(pid, v):
return redirect(p.permalink)
def archiveorg(url):
try: requests.get(f'https://web.archive.org/save/{url}', headers={'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'}, timeout=100)
except: pass
def thumbnail_thread(pid):
db = db_session()
def expand_url(post_url, fragment_url):
if fragment_url.startswith("https://"):
return fragment_url
elif fragment_url.startswith("https://"):
@ -399,8 +341,6 @@ def thumbnail_thread(pid):
if x.status_code != 200:
db.close()
return
if x.headers.get("Content-Type","").startswith("text/html"):
soup=BeautifulSoup(x.content, 'lxml')
@ -408,15 +348,13 @@ def thumbnail_thread(pid):
thumb_candidate_urls=[]
meta_tags = [
"drama:thumbnail",
"themotte:thumbnail",
"twitter:image",
"og:image",
"thumbnail"
]
for tag_name in meta_tags:
tag = soup.find(
'meta',
attrs={
@ -501,40 +439,7 @@ def api_is_repost():
url = request.values.get('url')
if not url: abort(400)
for rd in ("://reddit.com", "://new.reddit.com", "://www.reddit.com", "://redd.it", "://libredd.it", "://teddit.net"):
url = url.replace(rd, "://old.reddit.com")
url = url.replace("nitter.net", "twitter.com").replace("old.reddit.com/gallery", "reddit.com/gallery").replace("https://youtu.be/", "https://youtube.com/watch?v=").replace("https://music.youtube.com/watch?v=", "https://youtube.com/watch?v=").replace("https://streamable.com/", "https://streamable.com/e/").replace("https://youtube.com/shorts/", "https://youtube.com/watch?v=").replace("https://mobile.twitter", "https://twitter").replace("https://m.facebook", "https://facebook").replace("m.wikipedia.org", "wikipedia.org").replace("https://m.youtube", "https://youtube").replace("https://www.youtube", "https://youtube").replace("https://www.twitter", "https://twitter").replace("https://www.instagram", "https://instagram").replace("https://www.tiktok", "https://tiktok")
if "/i.imgur.com/" in url: url = url.replace(".png", ".webp").replace(".jpg", ".webp").replace(".jpeg", ".webp")
elif "/media.giphy.com/" in url or "/c.tenor.com/" in url: url = url.replace(".gif", ".webp")
elif "/i.ibb.com/" in url: url = url.replace(".png", ".webp").replace(".jpg", ".webp").replace(".jpeg", ".webp").replace(".gif", ".webp")
if url.startswith("https://streamable.com/") and not url.startswith("https://streamable.com/e/"): url = url.replace("https://streamable.com/", "https://streamable.com/e/")
parsed_url = urlparse(url)
domain = parsed_url.netloc
if domain in ('old.reddit.com','twitter.com','instagram.com','tiktok.com'):
new_url = ParseResult(scheme="https",
netloc=parsed_url.netloc,
path=parsed_url.path,
params=parsed_url.params,
query=None,
fragment=parsed_url.fragment)
else:
qd = parse_qs(parsed_url.query)
filtered = {k: val for k, val in qd.items() if not k.startswith('utm_') and not k.startswith('ref_')}
new_url = ParseResult(scheme="https",
netloc=parsed_url.netloc,
path=parsed_url.path,
params=parsed_url.params,
query=urlencode(filtered, doseq=True),
fragment=parsed_url.fragment)
url = urlunparse(new_url)
url = urllib.parse.unparse(canonicalize_url2(url, httpsify=True))
if url.endswith('/'): url = url[:-1]
search_url = sql_ilike_clean(url)
@ -546,13 +451,100 @@ def api_is_repost():
if repost: return {'permalink': repost.permalink}
else: return {'permalink': ''}
def _do_antispam_submission_check(v:User, validated:validators.ValidatedSubmissionLike):
now = int(time.time())
cutoff = now - 60 * 60 * 24
similar_posts = g.db.query(Submission).filter(
Submission.author_id == v.id,
Submission.title.op('<->')(validated.title) < app.config["SPAM_SIMILARITY_THRESHOLD"],
Submission.created_utc > cutoff
).all()
if validated.url:
similar_urls = g.db.query(Submission).filter(
Submission.author_id == v.id,
Submission.url.op('<->')(validated.url) < app.config["SPAM_URL_SIMILARITY_THRESHOLD"],
Submission.created_utc > cutoff
).all()
else:
similar_urls = []
threshold = app.config["SPAM_SIMILAR_COUNT_THRESHOLD"]
if v.age_seconds >= (60 * 60 * 24 * 7): threshold *= 3
elif v.age_seconds >= (60 * 60 * 24): threshold *= 2
if max(len(similar_urls), len(similar_posts)) < threshold:
return
text = "Your account has been banned for **1 day** for the following reason:\n\n> Too much spam!"
send_repeatable_notification(v.id, text)
v.ban(reason="Spamming.", days=1)
for post in similar_posts + similar_urls:
post.is_banned = True
post.is_pinned = False
post.ban_reason = "AutoJanny"
g.db.add(post)
ma=ModAction(
user_id=AUTOJANNY_ID,
target_submission_id=post.id,
kind="ban_post",
_note="spam"
)
g.db.add(ma)
g.db.commit()
abort(403)
def _execute_domain_ban_check(parsed_url:ParseResult):
domain:str = parsed_url.netloc
domain_obj = get_domain(domain)
if not domain_obj:
domain_obj = get_domain(domain+parsed_url.path)
if not domain_obj: return
abort(403, f"Remove the {domain_obj.domain} link from your post and try again. {domain_obj.reason}")
def _duplicate_check(search_url:Optional[str]) -> Optional[werkzeug.wrappers.Response]:
if not search_url: return None
repost = g.db.query(Submission).filter(
func.lower(Submission.url) == search_url.lower(),
Submission.deleted_utc == 0,
Submission.is_banned == False
).first()
if repost and SITE != 'localhost':
return redirect(repost.permalink)
return None
def _duplicate_check2(
user_id:int,
validated_post:validators.ValidatedSubmissionLike) -> Optional[werkzeug.wrappers.Response]:
dup = g.db.query(Submission).filter(
Submission.author_id == user_id,
Submission.deleted_utc == 0,
Submission.title == validated_post.title,
Submission.url == validated_post.url,
Submission.body == validated_post.body
).one_or_none()
if dup and SITE != 'localhost':
return redirect(dup.permalink)
return None
@app.post("/submit")
# @app.post("/h/<sub>/submit")
@limiter.limit("1/second;2/minute;10/hour;50/day")
@auth_required
def submit_post(v, sub=None):
def error(error):
title:str = request.values.get("title", "")
body:str = request.values.get("body", "")
url:str = request.values.get("url", "")
if request.headers.get("Authorization") or request.headers.get("xhr"): abort(400, error)
SUBS = [x[0] for x in g.db.query(Sub.name).order_by(Sub.name).all()]
@ -560,13 +552,13 @@ def submit_post(v, sub=None):
if v.is_suspended: return error("You can't perform this action while banned.")
title = guarded_value("title", 1, MAX_TITLE_LENGTH)
title = sanitize_raw(title, allow_newlines=False, length_limit=MAX_TITLE_LENGTH)
url = guarded_value("url", 0, MAX_URL_LENGTH)
body = guarded_value("body", 0, MAX_BODY_LENGTH)
body = sanitize_raw(body, allow_newlines=True, length_limit=MAX_BODY_LENGTH)
try:
validated_post: validators.ValidatedSubmissionLike = \
validators.ValidatedSubmissionLike.from_flask_request(request,
allow_embedding=MULTIMEDIA_EMBEDDING_ENABLED,
)
except ValueError as e:
return error(str(e))
sub = request.values.get("sub")
if sub: sub = sub.replace('/h/','').replace('s/','')
@ -578,171 +570,27 @@ def submit_post(v, sub=None):
sub = sub[0]
if v.exiled_from(sub): return error(f"You're exiled from /h/{sub}")
else: sub = None
title_html = filter_emojis_only(title, graceful=True)
if len(title_html) > 1500: return error("Rendered title is too big!")
duplicate:Optional[werkzeug.wrappers.Response] = \
_duplicate_check(validated_post.repost_search_url)
if duplicate: return duplicate
embed = None
parsed_url:Optional[ParseResult] = validated_post.url_canonical
if parsed_url:
_execute_domain_ban_check(parsed_url)
if url:
for rd in ("://reddit.com", "://new.reddit.com", "://www.reddit.com", "://redd.it", "://libredd.it", "://teddit.net"):
url = url.replace(rd, "://old.reddit.com")
duplicate:Optional[werkzeug.wrappers.Response] = \
_duplicate_check2(v.id, validated_post)
if duplicate: return duplicate
url = url.replace("nitter.net", "twitter.com").replace("old.reddit.com/gallery", "reddit.com/gallery").replace("https://youtu.be/", "https://youtube.com/watch?v=").replace("https://music.youtube.com/watch?v=", "https://youtube.com/watch?v=").replace("https://streamable.com/", "https://streamable.com/e/").replace("https://youtube.com/shorts/", "https://youtube.com/watch?v=").replace("https://mobile.twitter", "https://twitter").replace("https://m.facebook", "https://facebook").replace("m.wikipedia.org", "wikipedia.org").replace("https://m.youtube", "https://youtube").replace("https://www.youtube", "https://youtube").replace("https://www.twitter", "https://twitter").replace("https://www.instagram", "https://instagram").replace("https://www.tiktok", "https://tiktok")
if "/i.imgur.com/" in url: url = url.replace(".png", ".webp").replace(".jpg", ".webp").replace(".jpeg", ".webp")
elif "/media.giphy.com/" in url or "/c.tenor.com/" in url: url = url.replace(".gif", ".webp")
elif "/i.ibb.com/" in url: url = url.replace(".png", ".webp").replace(".jpg", ".webp").replace(".jpeg", ".webp").replace(".gif", ".webp")
if url.startswith("https://streamable.com/") and not url.startswith("https://streamable.com/e/"): url = url.replace("https://streamable.com/", "https://streamable.com/e/")
parsed_url = urlparse(url)
domain = parsed_url.netloc
if domain in ('old.reddit.com','twitter.com','instagram.com','tiktok.com'):
new_url = ParseResult(scheme="https",
netloc=parsed_url.netloc,
path=parsed_url.path,
params=parsed_url.params,
query=None,
fragment=parsed_url.fragment)
else:
qd = parse_qs(parsed_url.query)
filtered = {k: val for k, val in qd.items() if not k.startswith('utm_') and not k.startswith('ref_')}
new_url = ParseResult(scheme="https",
netloc=parsed_url.netloc,
path=parsed_url.path,
params=parsed_url.params,
query=urlencode(filtered, doseq=True),
fragment=parsed_url.fragment)
search_url = urlunparse(new_url)
if search_url.endswith('/'): url = url[:-1]
repost = g.db.query(Submission).filter(
func.lower(Submission.url) == search_url.lower(),
Submission.deleted_utc == 0,
Submission.is_banned == False
).first()
if repost and SITE != 'localhost': return redirect(repost.permalink)
domain_obj = get_domain(domain)
if not domain_obj: domain_obj = get_domain(domain+parsed_url.path)
if domain_obj:
reason = f"Remove the {domain_obj.domain} link from your post and try again. {domain_obj.reason}"
return error(reason)
elif "twitter.com" == domain:
try: embed = requests.get("https://publish.twitter.com/oembed", params={"url":url, "omit_script":"t"}, timeout=5).json()["html"]
except: pass
elif url.startswith('https://youtube.com/watch?v='):
url = unquote(url).replace('?t', '&t')
yt_id = url.split('https://youtube.com/watch?v=')[1].split('&')[0].split('%')[0]
if yt_id_regex.fullmatch(yt_id):
req = requests.get(f"https://www.googleapis.com/youtube/v3/videos?id={yt_id}&key={YOUTUBE_KEY}&part=contentDetails", timeout=5).json()
if req.get('items'):
params = parse_qs(urlparse(url).query)
t = params.get('t', params.get('start', [0]))[0]
if isinstance(t, str): t = t.replace('s','')
embed = f'<lite-youtube videoid="{yt_id}" params="autoplay=1&modestbranding=1'
if t:
try: embed += f'&start={int(t)}'
except: pass
embed += '"></lite-youtube>'
elif app.config['SERVER_NAME'] in domain and "/post/" in url and "context" not in url:
id = url.split("/post/")[1]
if "/" in id: id = id.split("/")[0]
embed = str(int(id))
if not url and not body and not request.files.get("file") and not request.files.get("file2"):
return error("Please enter a url or some text.")
dup = g.db.query(Submission).filter(
Submission.author_id == v.id,
Submission.deleted_utc == 0,
Submission.title == title,
Submission.url == url,
Submission.body == body
).one_or_none()
if dup and SITE != 'localhost': return redirect(dup.permalink)
now = int(time.time())
cutoff = now - 60 * 60 * 24
similar_posts = g.db.query(Submission).filter(
Submission.author_id == v.id,
Submission.title.op('<->')(title) < app.config["SPAM_SIMILARITY_THRESHOLD"],
Submission.created_utc > cutoff
).all()
if url:
similar_urls = g.db.query(Submission).filter(
Submission.author_id == v.id,
Submission.url.op('<->')(url) < app.config["SPAM_URL_SIMILARITY_THRESHOLD"],
Submission.created_utc > cutoff
).all()
else: similar_urls = []
threshold = app.config["SPAM_SIMILAR_COUNT_THRESHOLD"]
if v.age >= (60 * 60 * 24 * 7): threshold *= 3
elif v.age >= (60 * 60 * 24): threshold *= 2
if max(len(similar_urls), len(similar_posts)) >= threshold:
text = "Your account has been banned for **1 day** for the following reason:\n\n> Too much spam!"
send_repeatable_notification(v.id, text)
v.ban(reason="Spamming.",
days=1)
for post in similar_posts + similar_urls:
post.is_banned = True
post.is_pinned = False
post.ban_reason = "AutoJanny"
g.db.add(post)
ma=ModAction(
user_id=AUTOJANNY_ID,
target_submission_id=post.id,
kind="ban_post",
_note="spam"
)
g.db.add(ma)
return redirect("/notifications")
if request.files.get("file2") and request.headers.get("cf-ipcountry") != "T1":
files = request.files.getlist('file2')[:4]
for file in files:
if file.content_type.startswith('image/'):
name = f'/images/{time.time()}'.replace('.','') + '.webp'
file.save(name)
image = process_image(name)
if app.config['MULTIMEDIA_EMBEDDING_ENABLED']:
body += f"\n\n![]({image})"
else:
body += f'\n\n<a href="{image}">{image}</a>'
else:
return error("Image files only")
body_html = sanitize(body)
_do_antispam_submission_check(v, validated_post)
club = bool(request.values.get("club",""))
if embed and len(embed) > 1500: embed = None
is_bot = bool(request.headers.get("Authorization"))
# Invariant: these values are guarded and obey the length bound
assert len(title) <= MAX_TITLE_LENGTH
assert len(body) <= MAX_BODY_LENGTH
assert len(validated_post.title) <= MAX_TITLE_LENGTH
assert len(validated_post.body) <= SUBMISSION_BODY_LENGTH_MAXIMUM
post = Submission(
private=bool(request.values.get("private","")),
@ -750,75 +598,28 @@ def submit_post(v, sub=None):
author_id=v.id,
over_18=bool(request.values.get("over_18","")),
app_id=v.client.application.id if v.client else None,
is_bot = is_bot,
url=url,
body=body,
body_html=body_html,
embed_url=embed,
title=title,
title_html=title_html,
is_bot=is_bot,
url=validated_post.url,
body=validated_post.body,
body_html=validated_post.body_html,
embed_url=validated_post.embed_slow,
title=validated_post.title,
title_html=validated_post.title_html,
sub=sub,
ghost=False,
filter_state='filtered' if v.admin_level == 0 and app.config['SETTINGS']['FilterNewPosts'] else 'normal'
filter_state='filtered' if v.admin_level == 0 and app.config['SETTINGS']['FilterNewPosts'] else 'normal',
thumburl=validated_post.thumburl
)
g.db.add(post)
g.db.flush()
vote = Vote(user_id=v.id,
vote_type=1,
submission_id=post.id
)
g.db.add(vote)
if request.files.get('file') and request.headers.get("cf-ipcountry") != "T1":
file = request.files['file']
if file.content_type.startswith('image/'):
name = f'/images/{time.time()}'.replace('.','') + '.webp'
file.save(name)
post.url = process_image(name)
name2 = name.replace('.webp', 'r.webp')
copyfile(name, name2)
post.thumburl = process_image(name2, resize=100)
else:
return error("Image files only")
post.submit(g.db)
if not post.thumburl and post.url:
gevent.spawn(thumbnail_thread, post.id)
if not post.private and not post.ghost:
notify_users = NOTIFY_USERS(f'{title} {body}', v)
if notify_users:
cid = notif_comment2(post)
for x in notify_users:
add_notif(cid, x)
if (request.values.get('followers') or is_bot) and v.followers:
text = f"@{v.username} has made a new post: [{post.title}]({post.shortlink})"
if post.sub: text += f" in <a href='/h/{post.sub}'>/h/{post.sub}"
cid = notif_comment(text, autojanny=True)
for follow in v.followers:
user = get_account(follow.user_id)
if post.club and not user.paid_dues: continue
add_notif(cid, user.id)
v.post_count = g.db.query(Submission.id).filter_by(author_id=v.id, is_banned=False, deleted_utc=0).count()
g.db.add(v)
post.publish()
g.db.commit()
cache.delete_memoized(frontlist)
cache.delete_memoized(User.userpagelisting)
if v.admin_level > 0 and ("[changelog]" in post.title.lower() or "(changelog)" in post.title.lower()) and not post.private:
cache.delete_memoized(changeloglist)
if request.headers.get("Authorization"): return post.json
if request.headers.get("Authorization"):
return post.json
else:
post.voted = 1
if 'megathread' in post.title.lower(): sort = 'new'
@ -830,7 +631,6 @@ def submit_post(v, sub=None):
@limiter.limit("1/second;30/minute;200/hour;1000/day")
@auth_required
def delete_post_pid(pid, v):
post = get_post(pid)
if post.author_id != v.id:
abort(403)
@ -841,7 +641,7 @@ def delete_post_pid(pid, v):
g.db.add(post)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()
@ -853,10 +653,11 @@ def delete_post_pid(pid, v):
def undelete_post_pid(pid, v):
post = get_post(pid)
if post.author_id != v.id: abort(403)
post.deleted_utc =0
post.deleted_utc = 0
g.db.add(post)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()

7
files/routes/reporting.py
View file

@ -1,9 +1,10 @@
from files.helpers.wrappers import *
from files.helpers.get import *
from flask import g
from files.__main__ import app, limiter
from os import path
from files.helpers.get import *
from files.helpers.sanitize import filter_emojis_only
from files.helpers.wrappers import *
@app.post("/report/post/<pid>")
@limiter.limit("1/second;30/minute;200/hour;1000/day")

7
files/routes/search.py
View file

@ -1,11 +1,10 @@
from files.helpers.wrappers import *
import re
from sqlalchemy import *
from flask import *
from files.__main__ import app
from files.helpers.contentsorting import apply_time_filter, sort_objects
from files.helpers.strings import sql_ilike_clean
from files.helpers.wrappers import *
from files.routes.importstar import *
valid_params=[
'author',

23
files/routes/settings.py
View file

@ -1,15 +1,14 @@
import os
from shutil import copyfile
from files.__main__ import app, limiter
from files.helpers.alerts import *
from files.helpers.caching import invalidate_cache
from files.helpers.config.const import *
from files.helpers.media import process_image
from files.helpers.sanitize import *
from files.helpers.const import *
from files.mail import *
from files.__main__ import app, cache, limiter
from .front import frontlist
import os
from files.helpers.sanitize import filter_emojis_only
from files.helpers.strings import sql_ilike_clean
from shutil import copyfile
import requests
from files.mail import *
tiers={
"(Paypig)": 1,
@ -186,7 +185,7 @@ def settings_profile_post(v):
if frontsize in {"15", "25", "50", "100"}:
v.frontsize = int(frontsize)
updated = True
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
else: abort(400)
defaultsortingcomments = request.values.get("defaultsortingcomments")
@ -263,7 +262,7 @@ def changelogsub(v):
v.changelogsub = not v.changelogsub
g.db.add(v)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()
if v.changelogsub: return {"message": "You have subscribed to the changelog!"}
@ -542,7 +541,7 @@ def settings_block_user(v):
target_id=user.id,
)
g.db.add(new_block)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()
return {"message": f"@{user.username} blocked."}
@ -556,7 +555,7 @@ def settings_unblock_user(v):
x = v.is_blocking(user)
if not x: abort(409)
g.db.delete(x)
cache.delete_memoized(frontlist)
invalidate_cache(frontlist=True)
g.db.commit()
return {"message": f"@{user.username} unblocked."}

39
files/routes/static.py
View file

@ -1,17 +1,19 @@
import calendar
import matplotlib.pyplot as plt
from sqlalchemy import func
from files.classes.award import AWARDS
from files.classes.badges import BadgeDef
from files.classes.mod_logs import ACTIONTYPES, ACTIONTYPES2
from files.helpers.alerts import *
from files.helpers.captcha import validate_captcha
from files.helpers.config.const import *
from files.helpers.config.environment import HCAPTCHA_SECRET, HCAPTCHA_SITEKEY
from files.helpers.media import process_image
from files.mail import *
from files.__main__ import app, limiter, mail
from files.helpers.alerts import *
from files.helpers.const import *
from files.helpers.captcha import validate_captcha
from files.classes.award import AWARDS
from sqlalchemy import func
from os import path
import calendar
import matplotlib.pyplot as plt
from files.classes.mod_logs import ACTIONTYPES, ACTIONTYPES2
from files.classes.badges import BadgeDef
import logging
from files.__main__ import app, cache, limiter # violates isort but used to prevent getting shadowed
@app.get('/logged_out/')
@app.get('/logged_out/<path:old>')
@ -31,13 +33,6 @@ def logged_out(old = ""):
return redirect(redirect_url)
@app.get("/marsey_list")
@cache.memoize(timeout=600, make_name=make_name)
def marsey_list():
marseys = [f"{x.name} : {x.tags}" for x in g.db.query(Marsey).order_by(Marsey.count.desc())]
return str(marseys).replace("'",'"')
@app.get('/sidebar')
@auth_desired
def sidebar(v):
@ -280,15 +275,13 @@ def api(v):
@app.get("/media")
@auth_desired
def contact(v):
return render_template("contact.html", v=v,
hcaptcha=app.config.get("HCAPTCHA_SITEKEY", ""))
return render_template("contact.html", v=v, hcaptcha=HCAPTCHA_SITEKEY)
@app.post("/send_admin")
@limiter.limit("1/second;2/minute;6/hour;10/day")
@auth_desired
def submit_contact(v: Optional[User]):
if not v and not validate_captcha(app.config.get("HCAPTCHA_SECRET", ""),
app.config.get("HCAPTCHA_SITEKEY", ""),
if not v and not validate_captcha(HCAPTCHA_SECRET, HCAPTCHA_SITEKEY,
request.values.get("h-captcha-response", "")):
abort(403, "CAPTCHA provided was not correct. Please try it again")
body = request.values.get("message")

33
files/routes/users.py
View file

@ -1,27 +1,33 @@
import qrcode
import io
import time
import math
import time
from collections import Counter
from urllib.parse import urlparse
from files.classes.leaderboard import SimpleLeaderboard, BadgeMarseyLeaderboard, UserBlockLeaderboard, LeaderboardMeta
import gevent
import qrcode
import files.helpers.listing as listings
from files.__main__ import app, cache, limiter
from files.classes.leaderboard import (BadgeMarseyLeaderboard, LeaderboardMeta,
SimpleLeaderboard, UserBlockLeaderboard)
from files.classes.views import ViewerRelationship
from files.helpers.alerts import *
from files.helpers.assetcache import assetcache_path
from files.helpers.config.const import *
from files.helpers.contentsorting import apply_time_filter, sort_objects
from files.helpers.media import process_image
from files.helpers.sanitize import *
from files.helpers.strings import sql_ilike_clean
from files.helpers.const import *
from files.helpers.assetcache import assetcache_path
from files.helpers.contentsorting import apply_time_filter, sort_objects
from files.mail import *
from flask import *
from files.__main__ import app, limiter
from collections import Counter
import gevent
from files.routes.importstar import *
# warning: do not move currently. these have import-time side effects but
# until this is refactored to be not completely awful, there's not really
# a better option.
from files.helpers.services import *
from files.helpers.services import *
@app.get("/@<username>/upvoters/<uid>/posts")
@admin_level_required(3)
@ -673,7 +679,7 @@ def u_username(username, v=None):
try: page = max(int(request.values.get("page", 1)), 1)
except: page = 1
ids = u.userpagelisting(site=SITE, v=v, page=page, sort=sort, t=t)
ids = listings.userpagelisting(u, site=SITE, v=v, page=page, sort=sort, t=t)
next_exists = (len(ids) > 25)
ids = ids[:25]
@ -862,9 +868,6 @@ def remove_follow(username, v):
return {"message": "Follower removed!"}
from urllib.parse import urlparse
import re
@app.get("/pp/<int:id>")
@app.get("/uid/<int:id>/pic")
@app.get("/uid/<int:id>/pic/profile")

16
files/routes/volunteer.py
View file

@ -1,18 +1,16 @@
from datetime import datetime, timedelta
from typing import Optional
import sqlalchemy
from flask import abort, g, render_template, request
import files.helpers.jinja2
import files.routes.volunteer_janitor
from files.__main__ import app
from files.classes.user import User
import files.helpers.jinja2
from files.helpers.wrappers import auth_required
from files.routes.volunteer_common import VolunteerDuty
import files.routes.volunteer_janitor
from flask import abort, render_template, g, request
from os import environ
import sqlalchemy
from typing import Optional
import pprint
@files.helpers.jinja2.template_function

29
files/routes/votes.py
View file

@ -1,10 +1,13 @@
from files.helpers.wrappers import *
from files.helpers.get import *
from files.helpers.const import *
from files.classes import *
from flask import *
from files.__main__ import app, limiter, cache
from os import environ
from files.__main__ import app, limiter
from files.classes.comment import Comment
from files.classes.submission import Submission
from files.classes.votes import CommentVote, Vote
from files.helpers.config.const import OWNER_ID
from files.helpers.config.environment import ENABLE_DOWNVOTES
from files.helpers.get import get_comment, get_post
from files.helpers.wrappers import admin_level_required, is_not_permabanned
from files.routes.importstar import *
@app.get("/votes")
@limiter.exempt
@ -14,8 +17,8 @@ def admin_vote_info_get(v):
if not link: return render_template("votes.html", v=v)
try:
if "t2_" in link: thing = get_post(int(link.split("t2_")[1]), v=v)
elif "t3_" in link: thing = get_comment(int(link.split("t3_")[1]), v=v)
if "t2_" in link: thing = get_post(link.split("t2_")[1], v=v)
elif "t3_" in link: thing = get_comment(link.split("t3_")[1], v=v)
else: abort(400)
except: abort(400)
@ -54,11 +57,11 @@ def admin_vote_info_get(v):
@is_not_permabanned
def api_vote_post(post_id, new, v):
# make sure we're allowed in (is this really necessary? I'm not sure)
# make sure this account is not a bot
if request.headers.get("Authorization"): abort(403)
# make sure new is valid
if new == "-1" and environ.get('DISABLE_DOWNVOTES') == '1': abort(403, "forbidden.")
if new == "-1" and not ENABLE_DOWNVOTES: abort(403)
if new not in ["-1", "0", "1"]: abort(400)
new = int(new)
@ -122,11 +125,11 @@ def api_vote_post(post_id, new, v):
@is_not_permabanned
def api_vote_comment(comment_id, new, v):
# make sure we're allowed in (is this really necessary? I'm not sure)
# make sure this account is not a bot
if request.headers.get("Authorization"): abort(403)
# make sure new is valid
if new == "-1" and environ.get('DISABLE_DOWNVOTES') == '1': abort(403, "forbidden.")
if new == "-1" and not ENABLE_DOWNVOTES: abort(403)
if new not in ["-1", "0", "1"]: abort(400)
new = int(new)

87
files/templates/admin/admin_home.html
View file

@ -6,9 +6,7 @@
{% endblock %}
{% block content %}
<pre></pre>
<pre></pre>
<h3>Admin Tools</h3>
<h2 class="mt-2">Admin Tools</h2>
<div class="mb-3">
<strong>Users Here Now:</strong> {{g.loggedin_counter + g.loggedout_counter}} &mdash;
@ -16,52 +14,60 @@
<a href="/admin/loggedout">{{g.loggedout_counter}} Logged-Out</a>
</div>
<h4>Content</h4>
<h3>Content</h3>
<ul>
<li><a href="/admin/image_posts">Image Posts</a></li>
<li><a href="/admin/reported/posts">Reported Posts/Comments</a></li>
<li><a href="/admin/removed/posts">Removed Posts/Comments</a></li>
</ul>
<h4>Filtering</h4>
<h3>Filtering</h3>
<ul>
<li><a href="/admin/filtered/posts">Filtered Posts/Comments</a></li>
</ul>
<h4>Users</h4>
<h3>Users</h3>
<ul>
<li><a href="/admin/users">Users Feed</a></li>
<li><a href="/admin/shadowbanned">Shadowbanned Users</a></li>
<li><a href="/banned">Permabanned Users</a></li>
</ul>
<h4>Safety</h4>
<h3>Safety</h3>
<ul>
<li><a href="/admin/banned_domains">Banned Domains</a></li>
<li><a href="/admin/alt_votes">Multi Vote Analysis</a></li>
</ul>
<h4>Grant</h4>
<h3>Grant</h3>
<ul>
<li><a href="/admin/badge_grant">Grant Badges</a></li>
<li><a href="/admin/badge_remove">Remove Badges</a></li>
</ul>
<h4>API Access Control</h4>
<h3>API Access Control</h3>
<ul>
<li><a href="/admin/apps">Apps</a></li>
</ul>
<h4>Statistics</h4>
<h3>Statistics</h3>
<ul>
<li><a href="/stats">Content Stats</a></li>
<li><a href="/weekly_chart">Weekly Stat Chart</a></li>
<li><a href="/daily_chart">Daily Stat Chart</a></li>
</ul>
<section id="admin-section-scheduler" class="admin-section mt-3">
<h3>Scheduler</h3>
<ul>
{%- if v.admin_level >= PERMS['SCHEDULER'] -%}<li><a href="/tasks/">Tasks</a></li>{%- endif -%}
{%- if v.admin_level >= PERMS['SCHEDULER_POSTS'] -%}<li><a href="/tasks/scheduled_posts">Scheduled Posts</a></li>{%- endif -%}
</ul>
</section>
{% if v.admin_level >= 3 %}
<section id="admin-section-performance" class="admin-section mt-3">
<h4>Performance</h4>
<h3>Performance</h3>
<ul>
<li><a href="/performance/">Performance</a></li>
</ul>
@ -69,48 +75,33 @@
{% endif %}
{% if v.admin_level >= 3 %}
<pre></pre>
<div class="custom-control custom-switch">
<input autocomplete="off" type="checkbox" class="custom-control-input" id="signups" {% if site_settings['Signups'] %}checked{% endif %} onchange="post_toast(this,'/admin/site_settings/Signups');">
<label class="custom-control-label" for="signups">Signups</label>
</div>
<div class="custom-control custom-switch">
<input autocomplete="off" type="checkbox" class="custom-control-input" id="bots" {% if site_settings['Bots'] %}checked{% endif %} onchange="post_toast(this,'/admin/site_settings/Bots');">
<label class="custom-control-label" for="bots">Bots</label>
</div>
<div class="custom-control custom-switch">
<input autocomplete="off" type="checkbox" class="custom-control-input" id="FilterNewPosts" {% if site_settings['FilterNewPosts'] %}checked{% endif %} onchange="post_toast(this,'/admin/site_settings/FilterNewPosts');">
<label class="custom-control-label" for="FilterNewPosts">Filter New Posts</label>
</div>
<div class="custom-control custom-switch">
<input autocomplete="off" type="checkbox" class="custom-control-input" id="Read-only mode" {% if site_settings['Read-only mode'] %}checked{% endif %} onchange="post_toast(this,'/admin/site_settings/Read-only mode');">
<label class="custom-control-label" for="Read-only mode">Read-only mode</label>
</div>
<h3>Site Settings</h3>
{%- macro site_setting_bool(name, label) -%}
<div class="custom-control custom-switch">
<input autocomplete="off" type="checkbox" class="custom-control-input" id="site-setting-{{name}}" {% if site_settings[name] %}checked{% endif %} onchange="post_toast(this,'/admin/site_settings/{{name}}');">
<label class="custom-control-label" for="site-setting-{{name}}">{{label}}</label>
</div>
{%- endmacro -%}
{%- macro site_setting_int(name, label) -%}
<div class="custom-control custom-switch">
<input type="number" class="" id="site-setting-{{name}}" name="{{name}}" value="{{ site_settings[name] }}"
onblur="post_toast(null, '/admin/site_settings/{{name}}', false, { new_value: this.value })"/>
<label for="site-setting-{{name}}">{{label}}</label>
</div>
{%- endmacro -%}
{{site_setting_bool('signups', 'Signups')}}
{{site_setting_bool('bots', 'Bots')}}
{{site_setting_bool('FilterNewPosts', 'Filter New Posts')}}
{{site_setting_bool('Read-only mode', 'Read-Only Mode')}}
<div class="custom-control custom-switch">
<input autocomplete="off" type="checkbox" class="custom-control-input" id="under_attack" name="under_attack" {% if under_attack%}checked{% endif %} onchange="post_toast(this,'/admin/under_attack');">
<label class="custom-control-label" for="under_attack">Under attack mode</label>
</div>
<h4>Comment Filtering</h4>
<div class="custom-control custom-switch">
<input type="number" class="" id="min_comments" name="min_comments" value="{{ site_settings['FilterCommentsMinComments'] }}"
onblur="post_toast(null, '/admin/site_settings/FilterCommentsMinComments', false, { new_value: this.value })"/>
<label for="min_comments">Minimum Comments</label>
</div>
<div class="custom-control custom-switch">
<input type="number" class="" id="min_karma" name="min_karma" value="{{ site_settings['FilterCommentsMinKarma'] }}"
onblur="post_toast(null, '/admin/site_settings/FilterCommentsMinKarma', false, { new_value: this.value })"/>
<label for="min_karma">Minimum Karma</label>
</div>
<div class="custom-control custom-switch">
<input type="number" class="" id="min_age" name="min_age" value="{{ site_settings['FilterCommentsMinAgeDays'] }}"
onblur="post_toast(null, '/admin/site_settings/FilterCommentsMinAgeDays', false, { new_value: this.value })"/>
<label for="min_age">Minimum Account Age (days)</label>
</div>
<h4>Comment Filtering</h4>
{{site_setting_int('FilterCommentsMinComments', 'Minimum Comments')}}
{{site_setting_int('FilterCommentsMinKarma', 'Minimum Karma')}}
{{site_setting_int('FilterCommentsMinAgeDays', 'Minimum Account Age (Days)')}}
<button class="btn btn-primary mt-3" onclick="post_toast(this,'/admin/purge_cache');">PURGE CDN CACHE</button>
<button class="btn btn-primary mt-3" onclick="post_toast(this,'/admin/dump_cache');">DUMP INTERNAL CACHE</button>

2
files/templates/admin/alt_votes.html
View file

@ -74,7 +74,7 @@
<a role="button" class="btn btn-secondary" onclick="document.getElementById('linkbtn').classList.toggle('d-none');">Link Accounts</a>
<form action="/admin/link_accounts" method="post">
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<input type="hidden" name="u1" value="{{u1.id}}">
<input type="hidden" name="u2" value="{{u2.id}}">
<input type="submit" id="linkbtn" class="btn btn-primary d-none" value="Confirm Link: {{u1.username}} and {{u2.username}}">

2
files/templates/admin/app.html
View file

@ -19,7 +19,7 @@
<div class="body w-lg-100">
<label for="edit-{{app.id}}-author" class="mb-0 w-lg-25">User</label>
<input autocomplete="off" id="edit-{{app.id}}-author" class="form-control" type="text" name="name" value="{{app.author.username}}" readonly=readonly>
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<label for="edit-{{app.id}}-name" class="mb-0 w-lg-25">App Name</label>
<input autocomplete="off" id="edit-{{app.id}}-name" class="form-control" type="text" name="name" value="{{app.app_name}}" readonly=readonly>

2
files/templates/admin/awards.html
View file

@ -36,7 +36,7 @@
<h5>User Award Grant</h5>
<form action="/admin/awards", method="post">
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<label for="input-username">Username</label><br>
<input autocomplete="off" id="input-username" class="form-control mb-3" type="text" name="username" required>

2
files/templates/admin/badge_grant.html
View file

@ -36,7 +36,7 @@
<h5>Badge Grant</h5>
<form action="/admin/badge_grant", method="post">
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<label for="input-username">Username</label><br>

2
files/templates/admin/badge_remove.html
View file

@ -36,7 +36,7 @@
<h5>Badge Remove</h5>
<form action="/admin/badge_remove", method="post">
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<label for="input-username">Username</label><br>

2
files/templates/admin/banned_domains.html
View file

@ -28,7 +28,7 @@
<form action="/admin/banned_domains" method="post">
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<input autocomplete="off" name="domain" placeholder="Enter domain here.." class="form-control" required>
<input autocomplete="off" name="reason" placeholder="Enter ban reason here.." oninput="document.getElementById('ban-submit').disabled=false" class="form-control">
<input autocomplete="off" id="ban-submit" type="submit" class="btn btn-primary" value="Toggle ban" disabled>

2
files/templates/admin/sidebar.html
View file

@ -26,7 +26,7 @@
<div class="body d-lg-flex">
<div class="w-lg-100">
<form id="profile-settings" action="/admin/sidebar" method="post">
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<textarea autocomplete="off" maxlength="10000" class="form-control rounded" id="bio-text" aria-label="With textarea" placeholder="Enter sidebar here..." rows="50" name="sidebar" form="profile-settings">{% if sidebar %}{{sidebar}}{% endif %}</textarea>
<div class="d-flex mt-2">

22
files/templates/admin/tasks/scheduled_post.html Normal file
View file

@ -0,0 +1,22 @@
{%- extends "submission.html" -%}
{%- block comment_section -%}
<section class="scheduled-post-go-back bordered-section" style="padding:1em;">
You are viewing a single scheduled post. View <a href="/tasks/scheduled_posts/">all scheduled posts</a>
{%- if v.admin_level >= PERMS['SCHEDULER'] %} or <a href="/tasks/{{p.id}}">the associated task</a>{%- endif -%}
</section>
{%- if v.admin_level >= PERMS['SCHEDULER'] -%}
<section class="scheduled-post-edit scheduler-edit mt-3">
<h5 class="mb-2">Edit Scheduled Post</h5>
<form action="/tasks/scheduled_posts/{{p.id}}/schedule" method="post">
{{forms.scheduled_post_time_selection_form(v, p)}}
</form>
</section>
{%- endif -%}
<section class="scheduled-post-submissions mt-3">
<h5 class="mb-2">Previous Task Runs</h5>
{%- with listing = p.submissions -%}
{%- include "submission_listing.html" -%}
{%- endwith -%}
</section>
{%- endblock -%}
{%- block comment_section2 -%}{%- endblock -%}

20
files/templates/admin/tasks/scheduled_posts.html Normal file
View file

@ -0,0 +1,20 @@
{%- extends 'default.html' -%}
{%- block content -%}
<section id="scheduled-post-listing" class="listing submission-listing mt-3 mb-3">
{%- include "submission_listing.html" -%}
</section>
<section id="scheduled-post-new" class="settings-section rounded mb-3 bordered-section" style="width:75%">
<h5>Submit a Scheduled Post</h5>
<p>
This tool allows you to submit a scheduled post. It will be submitted as you at the scheduled date and time, and will perform tasks as if a post was submitted manually (including notifying any subscribers and any post-submit actions).<br>
You may use <a href="https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior">Python strftime formatting</a> in the title of your submitted post if you want to include the date in your submitted post.
</p>
<form action="/tasks/scheduled_posts/" method="post">
{# (label, name, id, min_length=none, max_length=none, required=false) #}
{{forms.text_field("Title", "title", "title", min_length=0, max_length=SUBMISSION_TITLE_LENGTH_MAXIMUM, required=true)}}
{{forms.text_field("URL (optional)", "url", "url", min_length=0, max_length=2048, required=false)}}
{{forms.textarea_field("Body", "body", "body", min_length=0, max_length=SUBMISSION_BODY_LENGTH_MAXIMUM, required=false)}}
{{forms.scheduled_post_time_selection_form(v, none)}}
</form>
</section>
{%- endblock -%}

41
files/templates/admin/tasks/single_run.html Normal file
View file

@ -0,0 +1,41 @@
{%- extends "default.html" -%}
{%- block content -%}
<h1 class="mt-2">Run for <a href="/tasks/{{run.task_id}}">Task #{{run.task_id}}</a></h1>
<section class="task-run-section task-{{run.task_id}} task-run-{{run.id}} mt-2" id="task-run-info-section">
<div class="overflow-x-auto">
<table class="table table-striped mb-5">
<tbody>
<tr>
<td>ID</td>
<td>{{run.id}}</td>
</tr>
<tr>
<td>Status</td>
<td>{{run.status_text}}</td>
</tr>
<tr>
<td>Started (UTC)</td>
<td>{{run.created_datetime}} ({{run.age_string}})</td>
</tr>
{%- if run.completed_utc -%}
<tr>
<td>Completed (UTC)</td>
<td>{{run.completed_datetime_str}} ({{run.completed_utc | agestamp}})</td>
</tr>
<tr>
<td>Elapsed Time</td>
<td>{{run.time_elapsed_str}}</td>
</tr>
{%- endif -%}
</tbody>
</table>
</div>
</section>
{%- if run.traceback_str and v.admin_level >= PERMS['SCHEDULER_TASK_TRACEBACK'] -%}
<section class="task-run-section task-{{run.task_id}} task-run-{{run.id}} mt-2" id="task-run-exception-section">
<h2>Exception Traceback</h2>
<p>During this run, the task encountered an unhandled exception.</p>
<pre class="mt-2">{{run.traceback_str}}</pre>
</section>
{%- endif -%}
{%- endblock -%}

91
files/templates/admin/tasks/single_task.html Normal file
View file

@ -0,0 +1,91 @@
{%- extends "default.html" -%}
{%- block content -%}
<h1 class="mt-3">Task #{{task.id}}</h1>
<section class="scheduled-post-go-back bordered-section" style="padding:1em;">
You are viewing a single task. View <a href="/tasks/">all tasks</a>
</section>
<section class="task-section task-{{task.id}} mt-3" id="task-info-section">
<div class="overflow-x-auto">
<table class="table table-striped mb-5">
<tbody>
<tr>
<td>ID</td>
<td>{{task.id}}</td>
</tr>
<tr>
<td>Status</td>
<td>{{task.run_state_enum.name.title()}}</td>
</tr>
<tr>
<td>Created (UTC)</td>
<td>{{task.created_datetime}} ({{task.age_string}})</td>
</tr>
<tr>
<td>Last Run (UTC)</td>
<td>{{task.run_time_last_str}}</td>
</tr>
<tr>
<td>Next Run (UTC)</td>
<td>{{task.next_trigger(task.run_time_last_or_created_utc) | timestamp}}</td>
</tr>
{%- if task.type == ScheduledTaskType.SCHEDULED_SUBMISSION -%}
<tr>
<td>Scheduled Post</td>
<td>
{%- if v.admin_level >= PERMS['SCHEDULER_POSTS'] -%}
<a href="/tasks/scheduled_posts/{{task.id}}">{{task.title}}</a>
{%- else -%}
{{task.title}}
{%- endif -%}
</td>
</tr>
{%- elif task.type == ScheduledTaskType.PYTHON_CALLABLE -%}
<tr>
<td>Import Path</td>
<td>{{task.import_path}}</td>
</tr>
<tr>
<td>Callable</td>
<td>{{task.callable}}</td>
</tr>
{%- endif -%}
<tr>
<td>Enabled</td>
<td>{{task.enabled}}</td>
</tr>
</tbody>
</table>
</div>
</section>
<section class="scheduled-task-edit scheduler-edit task-{{task.id}} mt-3" id="edit-schedule-section">
<h5 class="mb-2">Edit Scheduled Task</h5>
<form action="/tasks/{{task.id}}/schedule" method="post">
{{forms.scheduled_post_time_selection_form(v, task)}}
</form>
</section>
<section class="scheduled-task-run scheduler-edit task-{{task.id}} mt-3" id="task-run-section">
<h5 class="mb-2">Previous Task Runs</h5>
<div class="overflow-x-auto">
<table class="table table-striped mb-5">
<thead>
<tr>
<td>ID</td>
<td>Status</td>
<td>Started</td>
<td>Completed</td>
</tr>
</thead>
<tbody>
{%- for run in task.runs -%}
<tr>
<td><a href="/tasks/{{task.id}}/runs/{{run.id}}">{{run.id}}</a></td>
<td>{{run.status_text}}</td>
<td>{{run.age_string}}</td>
<td>{{run.completed_utc | agestamp}}</td>
</tr>
{%- endfor -%}
</tbody>
</table>
</div>
</section>
{%- endblock -%}

30
files/templates/admin/tasks/tasks.html Normal file
View file

@ -0,0 +1,30 @@
{%- extends "default.html" -%}
{%- block content -%}
<h1 class="mt-3">Scheduled Tasks</h1>
<section class="task-list-section mt-3" id="task-list-section">
<div class="overflow-x-auto">
<table class="table table-striped mb-5">
<thead>
<tr>
<td>Task</td>
<td>Created</td>
<td>Type</td>
<td>Status</td>
<td>Enabled</td>
</tr>
</thead>
<tbody>
{%- for task in listing -%}
<tr>
<td><a href="/tasks/{{task.id}}">Task {{task.id}}</a></td>
<td>{{task.age_string}}</td>
<td>{{task.type}}</td>
<td>{{task.run_state_enum.name.title()}}</td>
<td>{{task.enabled}}</td>
</tr>
{%- endfor -%}
</tbody>
</table>
</div>
</section>
{%- endblock -%}

2
files/templates/chat.html
View file

@ -152,7 +152,7 @@
<a class="{% if same %}d-none{% endif %} font-weight-bold text-black userlink" style="color:#{{m['namecolor']}}" target="_blank" href="/@{{m['username']}}">{{m['username']}}</a>
{% if not same %}
<span class="text-black time ml-2">{{m['time'] | timestamp}}</a>
<span class="text-black time ml-2">{{m['time'] | agestamp}}</a>
{% endif %}
<div class="cdiv">

2
files/templates/component/comment/voting_desktop.html
View file

@ -19,7 +19,7 @@
{% if voted == -1 %}
<li class=" arrow-down py-0 m-0 px-3 comment-{{c.id}}-down active"></li>
{% endif %}
{%- elif environ.get('DISABLE_DOWNVOTES') == '1' -%}
{%- elif not ENABLE_DOWNVOTES -%}
{# downvotes not allowed, nop #}
{%- elif v -%}
<button tabindex="0" role="button" onclick="vote('comment', '{{c.id}}', '-1')" class="comment-{{c.id}}-down btn caction py-0 m-0 px-3 nobackground arrow-down downvote-button comment-{{c.id}}-down {% if voted==-1 %}active{% endif %}"></button>

2
files/templates/component/comment/voting_mobile.html
View file

@ -13,7 +13,7 @@
<li id="voting-{{c.id}}-mobile" class="voting list-inline-item d-md-none">
<span tabindex="0" role="button" onclick="vote('comment-mobile', '{{c.id}}', '1')" class="comment-mobile-{{c.id}}-up mx-0 pr-1 arrow-up upvote-button comment-{{c.id}}-up {% if voted==1 %}active{% endif %}"></span>
<span class="comment-mobile-score-{{c.id}} score comment-score-{{c.id}} {% if voted==1 %}score-up{% elif voted==-1%}score-down{% endif %}{% if c.controversial %} controversial{% endif %}"{% if not c.is_banned %} data-bs-toggle="tooltip" data-bs-placement="top" title="+{{ups}} | -{{downs}}"{% endif %}>{{score}}</span>
<span {% if environ.get('DISABLE_DOWNVOTES') == '1' %}style="display: none!important"{% endif %} tabindex="0" role="button" onclick="vote('comment-mobile', '{{c.id}}', '-1')" class="comment-mobile-{{c.id}}-down mx-0 pl-1 my-0 arrow-down downvote-button comment-{{c.id}}-down {% if voted==-1 %}active{% endif %}"></span>
<span {% if not ENABLE_DOWNVOTES %}style="display: none!important"{% endif %} tabindex="0" role="button" onclick="vote('comment-mobile', '{{c.id}}', '-1')" class="comment-mobile-{{c.id}}-down mx-0 pl-1 my-0 arrow-down downvote-button comment-{{c.id}}-down {% if voted==-1 %}active{% endif %}"></span>
</li>
{% else %}
<li id="voting-{{c.id}}-mobile" class="voting list-inline-item d-md-none">

2
files/templates/component/modal/award.html
View file

@ -12,7 +12,7 @@
<input type="hidden" name="formkey", value="{{v.formkey}}">
<div class="card-columns award-columns awards-wrapper">
{% for award in v.user_awards %}
<a role="button" id="{{award.kind}}" class="card" onclick="pick('{{award.kind}}', {{award.price}}*{{v.discount}} <= {{v.procoins}}, {{award.price}}*{{v.discount}} <= {{v.coins}})">
<a role="button" id="{{award.kind}}" class="card" onclick="pick('{{award.kind}}', {{award.price}} <= {{v.procoins}}, {{award.price}} <= {{v.coins}})">
<i class="{{award.icon}} {{award.color}}"></i>
<div class="pt-2" style="font-weight: bold; font-size: 14px; color:#E1E1E1">{{award.title}}</div>
<div class="text-muted"><span id="{{award.kind}}-owned">{{award.owned}}</span> owned</div>

3
files/templates/component/post/actions.html
View file

@ -1,3 +1,5 @@
{%- if p.is_real_submission -%}
{% if v and v.id==p.author_id and p.private %}
<form action="/publish/{{p.id}}" method="post">
<input type="hidden" name="formkey", value="{{v.formkey}}">
@ -100,3 +102,4 @@
{% endif %}
</ul>
{% endif %}
{%- endif -%} {# {%- if p.is_real_submission -%} #}

2
files/templates/component/post/actions_mobile.html
View file

@ -1,3 +1,4 @@
{%- if p.is_real_submission -%}
{% if v and v.id==p.author_id and p.private %}
<form class="btn-block" action="/publish/{{p.id}}" method="post">
<input type="hidden" name="formkey", value="{{v.formkey}}">
@ -58,3 +59,4 @@
<button data-bs-dismiss="modal" id="unexile2" class="{% if not p.author.exiled_from(p.sub) %}d-none{% endif %} nobackground btn btn-link btn-block btn-lg text-left text-success" onclick="post_toast2(this,'/h/{{sub}}/unexile/{{p.author_id}}','exile2','unexile2')"><i class="fas fa-campfire mr-3 text-success"></i>Unexile user</button>
{% endif %}
{% endif %}
{%- endif -%} {# {%- if p.is_real_submission -%} #}

2
files/templates/contact.html
View file

@ -24,7 +24,7 @@
<input autocomplete="off" class="form-control" name="email" value="">
{% endif %}
<label for="input-message" class="mt-3">Your message</label>
<input type="hidden" name="formkey" value="{{v.formkey}}">
{{forms.formkey(v)}}
<textarea autocomplete="off" maxlength="{{MESSAGE_BODY_LENGTH_MAXIMUM}}" id="input-message" form="contactform" name="message" class="form-control" required></textarea>
<label class="btn btn-secondary m-0 mt-3" for="file-upload">
<div id="filename"><i class="far fa-image"></i></div>

1
files/templates/default.html
View file

@ -1,3 +1,4 @@
{%- import "util/forms.html" as forms -%}
<!DOCTYPE html>
<html lang="en">
<head>

Some files were not shown because too many files have changed in this diff Show more