fix(ops): sauvegarder reellement la base PostgreSQL
DATA-002 / OPS-001. backup.py ciblait SQLite : import sqlite3, DATABASE_PATH par defaut instance/team_tryouts.db, et l'API de sauvegarde sqlite3. La production tourne sur PostgreSQL, donc le fichier n'existait pas. Le script affichait "[WARNING] Database not found... Skipping database backup" -- puis, main() ne suivant que le resultat de la verification, **sortait avec le code 0**. Toute tache planifiee surveillant le code de sortie voyait vert alors qu'aucune sauvegarde n'avait jamais ete produite. Il n'existait donc aucune sauvegarde applicative de la base. Reecriture pg_dump en --format=custom : compresse, et pg_restore permet une restauration selective, ce qu'un dump SQL a plat ne permet pas. parse_database_url accepte les suffixes de dialecte SQLAlchemy (postgresql+psycopg://) que pg_dump ne comprend pas, et refuse explicitement une URL SQLite -- le cas exact qui passait en silence. Le mot de passe ne figure jamais dans la ligne de commande : il serait visible de tout processus capable de lister argv. Il passe par PGPASSWORD. Il est egalement absent des messages affiches, qui atterrissent dans les journaux du planificateur. verify_backup lit l'archive avec pg_restore --list et exige au moins une table : une archive illisible ne se restaure pas, et une archive sans table signifie que le dump a vise la mauvaise cible. Les deux sont des echecs silencieux qu'il vaut mieux attraper maintenant que pendant un incident. Le code de sortie vaut 0 uniquement si le dump a ete produit ET verifie. L'archive des documents est conservee : les contrats signes n'existent que sur disque, la base ne stocke que des chemins. Restaurer l'une sans l'autre laisse des lignes pointant vers des fichiers absents. docs/database-restore.md Procedure de restauration testable sur une base jetable, requetes de controle, demarrage de l'application sur la copie restauree, plan de reprise par scenario. ENABLE_DISCORD_BOT=false y est signale comme non optionnel : sans lui, l'exercice demarre un vrai bot et envoie de vraies notifications a de vraies personnes, a partir de donnees restaurees. Les points ouverts sont listes tels quels : aucune copie hors site, pas de chiffrement au repos, aucune planification, et l'exercice de restauration n'a jamais ete effectue. 17 tests sur ce qui est verifiable sans serveur PostgreSQL : analyse de l'URL, construction de la commande, non-fuite du mot de passe, et surtout codes de sortie -- le silence ne vaut plus succes. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
GGThed
+253
-108
@@ -1,84 +1,255 @@
|
||||
"""Database backup script for the Team Tryouts application.
|
||||
"""Database and document backup for the Team Tryouts application.
|
||||
|
||||
This module provides a simple backup mechanism for the SQLite database
|
||||
and uploaded contract documents. Designed to be run as a scheduled task
|
||||
(Windows Task Scheduler) or cron job.
|
||||
Dumps the PostgreSQL database with pg_dump and archives the uploaded
|
||||
contract documents. Designed to be run from a scheduled task (Windows Task
|
||||
Scheduler) or a cron job.
|
||||
|
||||
Usage:
|
||||
python backup.py
|
||||
|
||||
python app/supporting_scripts/backup.py
|
||||
python app/supporting_scripts/backup.py --verify-only <archive>
|
||||
|
||||
Configuration via environment variables:
|
||||
BACKUP_DIR: Directory to store backups (default: ./backups)
|
||||
BACKUP_RETENTION_DAYS: Number of days to keep backups (default: 30)
|
||||
DATABASE_URL PostgreSQL connection string (required)
|
||||
BACKUP_DIR Where to store backups (default: ./backups)
|
||||
BACKUP_RETENTION_DAYS How long to keep them (default: 30)
|
||||
PG_DUMP Path to pg_dump if not on PATH
|
||||
PG_RESTORE Path to pg_restore if not on PATH
|
||||
|
||||
A note on what this file used to be
|
||||
-----------------------------------
|
||||
The previous version targeted **SQLite**: it imported sqlite3, read
|
||||
DATABASE_PATH defaulting to instance/team_tryouts.db, and used the sqlite3
|
||||
backup API. Production runs on PostgreSQL, so the file never existed, the
|
||||
script printed "[WARNING] Database not found... Skipping database backup"
|
||||
and — because main() only tracked the verification result — still exited 0.
|
||||
It reported success while backing up nothing at all. Any scheduled task
|
||||
watching the exit code saw green.
|
||||
|
||||
Restoring is documented in docs/restauration-base.md. A backup that has
|
||||
never been restored is not a backup.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
# Configuration
|
||||
BACKUP_DIR = os.getenv('BACKUP_DIR', os.path.join(os.getcwd(), 'backups'))
|
||||
BACKUP_RETENTION_DAYS = int(os.getenv('BACKUP_RETENTION_DAYS', 30))
|
||||
DATABASE_PATH = os.getenv('DATABASE_PATH', os.path.join(os.getcwd(), 'instance', 'team_tryouts.db'))
|
||||
DOCUMENTS_DIR = os.path.join(os.getcwd(), 'documents')
|
||||
PG_DUMP = os.getenv('PG_DUMP', 'pg_dump')
|
||||
PG_RESTORE = os.getenv('PG_RESTORE', 'pg_restore')
|
||||
|
||||
|
||||
class BackupError(Exception):
|
||||
"""Raised when a backup step fails in a way that must stop the run."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Connection handling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_database_url(url):
|
||||
"""Split a SQLAlchemy/PostgreSQL URL into pg_dump connection settings.
|
||||
|
||||
Accepts the dialect suffixes SQLAlchemy uses (postgresql+psycopg://),
|
||||
which pg_dump does not understand.
|
||||
|
||||
Args:
|
||||
url: The connection string.
|
||||
|
||||
Returns:
|
||||
dict: host, port, dbname, user, password.
|
||||
|
||||
Raises:
|
||||
BackupError: If the URL is missing or is not a PostgreSQL one.
|
||||
"""
|
||||
if not url:
|
||||
raise BackupError('DATABASE_URL is not set.')
|
||||
|
||||
parsed = urlparse(url)
|
||||
scheme = parsed.scheme.split('+')[0]
|
||||
if scheme not in ('postgresql', 'postgres'):
|
||||
raise BackupError(
|
||||
f'DATABASE_URL is not a PostgreSQL connection string (scheme: {scheme!r}). '
|
||||
'This script only backs up PostgreSQL.'
|
||||
)
|
||||
|
||||
dbname = (parsed.path or '').lstrip('/')
|
||||
if not dbname:
|
||||
raise BackupError('DATABASE_URL does not name a database.')
|
||||
|
||||
return {
|
||||
'host': parsed.hostname or 'localhost',
|
||||
'port': str(parsed.port or 5432),
|
||||
'dbname': dbname,
|
||||
'user': unquote(parsed.username) if parsed.username else '',
|
||||
'password': unquote(parsed.password) if parsed.password else '',
|
||||
}
|
||||
|
||||
|
||||
def describe_target(conn):
|
||||
"""Human-readable target, deliberately without the password."""
|
||||
user = f'{conn["user"]}@' if conn['user'] else ''
|
||||
return f'{user}{conn["host"]}:{conn["port"]}/{conn["dbname"]}'
|
||||
|
||||
|
||||
def build_dump_command(conn, output_path):
|
||||
"""Assemble the pg_dump invocation.
|
||||
|
||||
--format=custom is compressed and lets pg_restore rebuild selectively;
|
||||
plain SQL would be larger and all-or-nothing.
|
||||
|
||||
The password is never placed on the command line — it would be visible
|
||||
to anyone able to list processes. It travels through PGPASSWORD instead,
|
||||
which is what pg_dump documents for non-interactive use.
|
||||
"""
|
||||
return [
|
||||
PG_DUMP,
|
||||
'--host', conn['host'],
|
||||
'--port', conn['port'],
|
||||
'--username', conn['user'],
|
||||
'--dbname', conn['dbname'],
|
||||
'--format=custom',
|
||||
'--no-owner',
|
||||
'--no-privileges',
|
||||
'--file', output_path,
|
||||
]
|
||||
|
||||
|
||||
def dump_environment(conn):
|
||||
"""Environment for pg_dump/pg_restore, carrying the password out of argv."""
|
||||
env = os.environ.copy()
|
||||
if conn['password']:
|
||||
env['PGPASSWORD'] = conn['password']
|
||||
return env
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backup steps
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def create_backup_dir():
|
||||
"""Create the backup directory if it doesn't exist."""
|
||||
os.makedirs(BACKUP_DIR, exist_ok=True)
|
||||
|
||||
|
||||
def backup_database():
|
||||
"""Backup the SQLite database using sqlite3's built-in backup API.
|
||||
|
||||
Returns:
|
||||
str: Path to the created backup file, or None if failed.
|
||||
"""
|
||||
if not os.path.exists(DATABASE_PATH):
|
||||
print(f'[WARNING] Database not found at {DATABASE_PATH}. Skipping database backup.')
|
||||
return None
|
||||
def backup_database(conn):
|
||||
"""Dump the PostgreSQL database.
|
||||
|
||||
Returns:
|
||||
str: Path to the created archive.
|
||||
|
||||
Raises:
|
||||
BackupError: If pg_dump is missing, fails, or produces nothing.
|
||||
"""
|
||||
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
|
||||
backup_filename = f'db_backup_{timestamp}.db'
|
||||
backup_path = os.path.join(BACKUP_DIR, backup_filename)
|
||||
backup_path = os.path.join(BACKUP_DIR, f'db_backup_{timestamp}.dump')
|
||||
|
||||
print(f'[INFO] Dumping {describe_target(conn)}')
|
||||
|
||||
try:
|
||||
source = sqlite3.connect(DATABASE_PATH)
|
||||
destination = sqlite3.connect(backup_path)
|
||||
source.backup(destination)
|
||||
source.close()
|
||||
destination.close()
|
||||
print(f'[OK] Database backed up to: {backup_path}')
|
||||
return backup_path
|
||||
except Exception as e:
|
||||
print(f'[ERROR] Database backup failed: {e}')
|
||||
return None
|
||||
result = subprocess.run(
|
||||
build_dump_command(conn, backup_path),
|
||||
env=dump_environment(conn),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=900,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
raise BackupError(
|
||||
f'{PG_DUMP} not found. Install the PostgreSQL client tools, or set '
|
||||
'PG_DUMP to its full path.'
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
raise BackupError('pg_dump timed out after 15 minutes.')
|
||||
|
||||
if result.returncode != 0:
|
||||
raise BackupError(f'pg_dump failed: {result.stderr.strip()}')
|
||||
|
||||
if not os.path.exists(backup_path) or os.path.getsize(backup_path) == 0:
|
||||
raise BackupError('pg_dump reported success but produced an empty file.')
|
||||
|
||||
size_mb = os.path.getsize(backup_path) / (1024 * 1024)
|
||||
print(f'[OK] Database backed up to: {backup_path} ({size_mb:.1f} MB)')
|
||||
return backup_path
|
||||
|
||||
|
||||
def verify_backup(backup_path):
|
||||
"""Check that the archive is readable and actually contains tables.
|
||||
|
||||
pg_restore --list parses the whole archive without touching any
|
||||
database. A dump that cannot be listed cannot be restored, and an
|
||||
archive holding no table would mean the dump ran against the wrong
|
||||
target — both are silent failures worth catching here rather than
|
||||
during an incident.
|
||||
|
||||
Args:
|
||||
backup_path: Path to the archive to verify.
|
||||
|
||||
Returns:
|
||||
bool: True if the archive looks restorable.
|
||||
"""
|
||||
if not backup_path or not os.path.exists(backup_path):
|
||||
print('[ERROR] Nothing to verify.')
|
||||
return False
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[PG_RESTORE, '--list', backup_path],
|
||||
capture_output=True, text=True, timeout=300,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
print(f'[WARNING] {PG_RESTORE} not found: archive left unverified.')
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
print('[ERROR] pg_restore --list timed out.')
|
||||
return False
|
||||
|
||||
if result.returncode != 0:
|
||||
print(f'[ERROR] Archive is not readable: {result.stderr.strip()}')
|
||||
return False
|
||||
|
||||
table_count = sum(1 for line in result.stdout.splitlines()
|
||||
if ' TABLE DATA ' in line)
|
||||
if table_count == 0:
|
||||
print('[ERROR] Archive contains no table data.')
|
||||
return False
|
||||
|
||||
print(f'[OK] Archive verified: {table_count} table(s) present.')
|
||||
return True
|
||||
|
||||
|
||||
def backup_documents():
|
||||
"""Backup the uploaded contract documents directory.
|
||||
|
||||
"""Archive the uploaded contract documents directory.
|
||||
|
||||
Returns:
|
||||
str: Path to the created archive, or None if no documents exist.
|
||||
str: Path to the created archive, or None if there is nothing to
|
||||
archive. Signed contracts live only on disk, so losing this
|
||||
directory loses the documents themselves.
|
||||
"""
|
||||
if not os.path.exists(DOCUMENTS_DIR):
|
||||
print('[INFO] No documents directory found. Skipping document backup.')
|
||||
return None
|
||||
|
||||
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
|
||||
archive_basename = f'documents_backup_{timestamp}'
|
||||
archive_path = os.path.join(BACKUP_DIR, archive_basename)
|
||||
archive_basename = os.path.join(BACKUP_DIR, f'documents_backup_{timestamp}')
|
||||
|
||||
try:
|
||||
shutil.make_archive(archive_path, 'zip', DOCUMENTS_DIR)
|
||||
zip_path = f'{archive_path}.zip'
|
||||
print(f'[OK] Documents backed up to: {zip_path}')
|
||||
return zip_path
|
||||
except Exception as e:
|
||||
print(f'[ERROR] Document backup failed: {e}')
|
||||
shutil.make_archive(archive_basename, 'zip', DOCUMENTS_DIR)
|
||||
except Exception as exc:
|
||||
print(f'[ERROR] Document backup failed: {exc}')
|
||||
return None
|
||||
|
||||
zip_path = f'{archive_basename}.zip'
|
||||
size_mb = os.path.getsize(zip_path) / (1024 * 1024)
|
||||
print(f'[OK] Documents backed up to: {zip_path} ({size_mb:.1f} MB)')
|
||||
return zip_path
|
||||
|
||||
|
||||
def cleanup_old_backups():
|
||||
"""Remove backup files older than BACKUP_RETENTION_DAYS."""
|
||||
@@ -90,95 +261,69 @@ def cleanup_old_backups():
|
||||
|
||||
for filename in os.listdir(BACKUP_DIR):
|
||||
file_path = os.path.join(BACKUP_DIR, filename)
|
||||
if os.path.isfile(file_path):
|
||||
file_time = datetime.fromtimestamp(os.path.getmtime(file_path))
|
||||
if file_time < cutoff:
|
||||
try:
|
||||
os.remove(file_path)
|
||||
removed_count += 1
|
||||
print(f'[CLEANUP] Removed old backup: {filename}')
|
||||
except OSError as e:
|
||||
print(f'[WARNING] Could not remove {filename}: {e}')
|
||||
if not os.path.isfile(file_path):
|
||||
continue
|
||||
if datetime.fromtimestamp(os.path.getmtime(file_path)) >= cutoff:
|
||||
continue
|
||||
try:
|
||||
os.remove(file_path)
|
||||
removed_count += 1
|
||||
print(f'[CLEANUP] Removed old backup: {filename}')
|
||||
except OSError as exc:
|
||||
print(f'[WARNING] Could not remove {filename}: {exc}')
|
||||
|
||||
if removed_count > 0:
|
||||
if removed_count:
|
||||
print(f'[CLEANUP] Removed {removed_count} old backup(s).')
|
||||
else:
|
||||
print('[CLEANUP] No old backups to remove.')
|
||||
|
||||
|
||||
def verify_backup(backup_path):
|
||||
"""Verify a database backup by running a quick integrity check.
|
||||
|
||||
Args:
|
||||
backup_path: Path to the backup file to verify.
|
||||
|
||||
Returns:
|
||||
bool: True if backup is valid, False otherwise.
|
||||
"""
|
||||
if not backup_path or not os.path.exists(backup_path):
|
||||
return False
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
try:
|
||||
conn = sqlite3.connect(backup_path)
|
||||
cursor = conn.cursor()
|
||||
cursor.execute('PRAGMA integrity_check')
|
||||
result = cursor.fetchone()
|
||||
conn.close()
|
||||
is_valid = result[0] == 'ok'
|
||||
if is_valid:
|
||||
print(f'[OK] Backup integrity verified: {backup_path}')
|
||||
else:
|
||||
print(f'[ERROR] Backup integrity check failed: {backup_path} - {result[0]}')
|
||||
return is_valid
|
||||
except Exception as e:
|
||||
print(f'[ERROR] Backup verification failed: {e}')
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
def main(argv=None):
|
||||
"""Run the full backup process.
|
||||
|
||||
Steps:
|
||||
1. Create backup directory
|
||||
2. Backup database
|
||||
3. Backup documents (if any)
|
||||
4. Verify database backup
|
||||
5. Clean up old backups
|
||||
|
||||
|
||||
Returns:
|
||||
int: 0 on success, 1 on failure.
|
||||
int: 0 when the database was dumped AND verified, 1 otherwise. The
|
||||
previous version returned 0 even when it had backed up nothing.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(description='Team Tryouts backup')
|
||||
parser.add_argument('--verify-only', metavar='ARCHIVE',
|
||||
help='Verify an existing archive and exit')
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if args.verify_only:
|
||||
return 0 if verify_backup(args.verify_only) else 1
|
||||
|
||||
print('=== Team Tryouts Backup ===')
|
||||
print(f'Started at: {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}')
|
||||
print(f'Backup directory: {BACKUP_DIR}')
|
||||
print(f'Retention period: {BACKUP_RETENTION_DAYS} days')
|
||||
print()
|
||||
|
||||
create_backup_dir()
|
||||
try:
|
||||
conn = parse_database_url(os.getenv('DATABASE_URL'))
|
||||
create_backup_dir()
|
||||
backup_path = backup_database(conn)
|
||||
except BackupError as exc:
|
||||
print(f'[ERROR] {exc}')
|
||||
print('\n=== Backup FAILED — no database backup was produced ===')
|
||||
return 1
|
||||
|
||||
# 1. Backup database
|
||||
db_backup_path = backup_database()
|
||||
success = True
|
||||
|
||||
# 2. Verify database backup
|
||||
if db_backup_path:
|
||||
if not verify_backup(db_backup_path):
|
||||
success = False
|
||||
|
||||
# 3. Backup documents
|
||||
verified = verify_backup(backup_path)
|
||||
backup_documents()
|
||||
|
||||
# 4. Cleanup old backups
|
||||
cleanup_old_backups()
|
||||
|
||||
print()
|
||||
if success:
|
||||
if verified:
|
||||
print('=== Backup completed successfully ===')
|
||||
else:
|
||||
print('=== Backup completed with warnings ===')
|
||||
return 0
|
||||
|
||||
return 0 if success else 1
|
||||
print('=== Backup FAILED verification — do not rely on this archive ===')
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
exit(main())
|
||||
sys.exit(main())
|
||||
|
||||
Reference in New Issue
Block a user