Flask Windows Service 6-Hour Forced Restart: After NSSM Saved 11 Days, PRO Died Again
Then Sept 19 a user emailed that PRO login wasn't working. I checked: PRO process was dead, watchdog bat file was gone, schtasks task was gone too.
The 11 days of "stable operation" was an illusion — the watchdog bat had been wiped by some Windows Defender scan weeks ago, I just didn't notice. Sept 23 I did one thing: trust no "automatic" mechanism, added a 6-hour forced restart fallback. Three-layer guardian: bat loop for instant recovery + schtasks 6-hour forced restart + manual one-click restart.
↓ 11 days stable (the "528 hours no failure" from W9)
Sept 19 PRO dies silently (I didn't notice)
↓ User report + investigation
Sept 23 add 6-hour forced restart + bat loop + monitoring alert
1. Last Time I Thought It Was Solved, But It Wasn't
On Sept 7 I spent 3.5 hours wrestling with NSSM, finally used schtasks + watchdog bat. When writing W9 I was proud: "528 hours no failure".
Then Sept 19 Sunday afternoon, a user emailed that PRO login was broken:
Subject: PRO login broken
Content: Hi, https://www.baccpc.com:8000/login has been loading forever
since yesterday. I'm a VIP member, please help.
I RDP'd in to check:
netstat -an | findstr ":8000"
# No output
Get-Process python
# No output
PRO Flask process was dead, and had been dead for at least 24 hours. PHD was still alive (still usable Sept 23 morning), but PRO was gone.
I immediately checked D:\phd824 and D:\20260516\vb_bendi_v24 for the bat files:
Test-Path D:\phd824\run_phd_loop.bat
False
Test-Path D:\20260516\vb_bendi_v24\run_pro_loop.bat
False
Both bats were gone. The watchdog bat was cleaned up between Sept 19 and Sept 23 by some person or system, schtasks task though still there, but the target bat path can't be found, so when service died no one pulled it back up.
2. Why the Watchdog Rescue Isn't Reliable
W9's logic: Python dies → bat detects process exit → bat loop goes to :restart → kill all python → restart Python.
Sounds perfect, but assumes the bat itself is alive. These 3 situations break it:
- Bat file deleted: Windows Defender / third-party AV / disk cleanup / manual misclick
- Bat process killed: AV treats bat as suspicious script, or cmd.exe itself gets recycled
- schtasks task fails: task disabled, deleted, or bat path not found
Any one of these breaks the entire guardian. Sept 19 I hit #1 (file wiped).
3. 6-Hour Forced Restart: Trust No "Automatic"
Sept 23 I redesigned: don't rely on any bat file staying alive, let the OS layer do scheduled kill + restart.
Simple idea:
- Layer 1 (second-level): bat loop, process dies → instant restart (same as W9, but bat itself moved to Desktop so it won't get cleaned)
- Layer 2 (hour-level): schtasks every 6 hours force taskkill python.exe, then Layer 1 bat pulls up new Python
- Layer 3 (day-level): server reboot (weekly / irregular) → Task Scheduler AtStartup auto-runs bat
Each layer works independently, any layer fails, the others fallback.
Layer 1: Bat Loop (Desktop Version, Not on Server)
I put the bat on Desktop, not in D:\ root or next to app.py, so Windows Defender disk cleanup scans of D:\ won't reach it:
Desktop `app.py -PRO.bat`:
@echo off
setlocal
title BaccAI PRO - 6h Auto Restart
cd /d D:\20260516\vb_bendi_v24
:loop
echo [%date% %time%] PRO watchdog started...
C:\Users\Administrator\AppData\Local\Programs\Python\Python38\python.exe app.py
goto :restart
:restart
echo [%date% %time%] PRO exited, killing all python and restarting in 10s...
timeout /t 10 /nobreak >nul
taskkill /F /IM python.exe /T 2>nul
timeout /t 5 /nobreak >nul
goto :loop
Desktop `app.py - PHD.bat` (same logic, path changed to D:\phd824).
Layer 2: schtasks 6-Hour Forced taskkill
Windows Task Scheduler doesn't need a bat running continuously, it's a Windows service maintained by the system. I configured 2 6-hourly scheduled tasks, every 6 hours force kill all python.exe:
schtasks /Create /TN "BaccAI_PRO_6h_restart" /TR "taskkill /F /IM python.exe /T" /SC HOURLY /MO 6 /RL HIGHEST /RU SYSTEM /F
schtasks /Create /TN "BaccAI_PHD_6h_restart" /TR "taskkill /F /IM python.exe /T" /SC HOURLY /MO 6 /RL HIGHEST /RU SYSTEM /F
Parameter explanation:
/TN= task name (BaccAI_PRO_6h_restart / BaccAI_PHD_6h_restart)/TR= command when triggered (taskkill /F /IM python.exe /T)/SC HOURLY /MO 6= trigger: every 6 hours/RL HIGHEST= highest privilege/RU SYSTEM= SYSTEM account (matches schtasks service itself, no bat needed)/F= force overwrite
Key design: schtasks itself depends on no bat file, it's a Windows service that never gets wiped by Defender. taskkill is a Windows built-in exe, also no bat dependency.
Layer 3: Server Reboot Auto-Recovery
If the server itself reboots (Windows Update / power loss), desktop bat also gets closed. I additionally configured 2 ONSTART tasks, auto-run desktop bat 30 seconds after boot:
schtasks /Create /TN "BaccAI_PRO_OnBoot" /TR "C:\Users\Administrator\Desktop\app.py -PRO.bat" /SC ONSTART /RL HIGHEST /RU SYSTEM /F
schtasks /Create /TN "BaccAI_PHD_OnBoot" /TR "C:\Users\Administrator\Desktop\app.py - PHD.bat" /SC ONSTART /RL HIGHEST /RU SYSTEM /F
One gotcha: ONSTART runs bat in session 0 (system service session), no window shown, no stdin received (killing via Ctrl+C is awkward). But we don't need to kill, just let it run.
4. 6 Hours Is the Sweet Spot
Why 6 hours, not 1 hour or 24 hours?
| Interval | Pros | Cons | Best for |
|---|---|---|---|
| 1 hour | Dead process instantly restored | 24 restarts per day, users lose login state / form data | Internal tools, no UX concern |
| 6 hours | 4 per day, acceptable UX | Short tasks may collide | ✅ Most web services, recommended |
| 24 hours | Users barely notice | Process quietly dead, 24h no recovery | Must-not-interrupt services (payment / medical) |
I chose 6 hours. Reasoning:
- 4 restarts per day, each with 15-20s downtime (taskkill + bat wait + Flask startup + model load)
- User collision probability: assuming 10 min daily access per user, hitting 15s window = 10/(6*60*60) ≈ 0.05%, near zero
- Even if collision happens, refresh fixes it — 100x better than "site down for hours"
5. Common Pitfalls + Fixes
Pitfall 1: Bat Window Accidentally Closed
My coworker occasionally closes the cmd window by mistake, watchdog dies. Fix:
- Change bat title to "BaccAI PRO - DO NOT CLOSE" so coworkers know not to close
- Or write a .vbs to launch bat (hides cmd window), but not recommended, harder to debug
Pitfall 2: schtasks Task Disabled
Windows Update or system optimization tools may "disable" schtasks tasks. Check:
schtasks /Query /TN "BaccAI_PRO_6h_restart"
Check the "Status" column, anything other than "Ready" is a problem.
Pitfall 3: taskkill Kills Other Python Processes
My server only runs these two Flask apps, so taskkill python.exe is safe. If you have other Python programs, filter by window title:
taskkill /F /FI "WINDOWTITLE eq BaccAI*" /T
Only kills windows with BaccAI in title.
Pitfall 4: Slow Model Loading
PRO's LSTM model loading takes 5-10 seconds, plus DB connection pool init, total 15-20 seconds. If user happens to be using PRO for prediction, restart loses their work.
Short-term compromise: 6 hours is short enough that user collision rate is low.
Long-term solution: Add Nginx reverse proxy, rolling restart (kill one worker, start new, kill next), zero-downtime. But cost is high, wait for W10+ traffic to justify.
6. Complete Start Card (D:\seo\START_COMMANDS.txt Updated)
Step 1: Status Check
netstat -an | findstr ":8443 :8000 :8080"
Get-Process python | Select Id, StartTime
schtasks /Query /TN "BaccAI_PRO_6h_restart"
schtasks /Query /TN "BaccAI_PHD_6h_restart"
Step 2: Start (4 Ways, Pick One)
:: Way 1: Double-click desktop bat (recommended, startup + watchdog auto-restart)
:: C:\Users\Administrator\Desktop\app.py -PRO.bat
:: C:\Users\Administrator\Desktop\app.py - PHD.bat
:: Way 2: Start-Process in background
Start-Process "C:\Users\Administrator\Desktop\app.py -PRO.bat"
Start-Process "C:\Users\Administrator\Desktop\app.py - PHD.bat"
:: Way 3: Manual one-time (for debugging)
cd /d D:\phd824
C:\Users\Administrator\AppData\Local\Programs\Python\Python38\python.exe app.py
Step 3: Emergency Restart (1 Command)
taskkill /F /IM python.exe /T
timeout /t 5 /nobreak
Start-Process "C:\Users\Administrator\Desktop\app.py -PRO.bat"
Start-Process "C:\Users\Administrator\Desktop\app.py - PHD.bat"
Step 4: 6-Hour Restart Tasks (First Deploy)
schtasks /Create /TN "BaccAI_PRO_6h_restart" /TR "taskkill /F /IM python.exe /T" /SC HOURLY /MO 6 /RL HIGHEST /RU SYSTEM /F
schtasks /Create /TN "BaccAI_PHD_6h_restart" /TR "taskkill /F /IM python.exe /T" /SC HOURLY /MO 6 /RL HIGHEST /RU SYSTEM /F
schtasks /Create /TN "BaccAI_PRO_OnBoot" /TR "C:\Users\Administrator\Desktop\app.py -PRO.bat" /SC ONSTART /RL HIGHEST /RU SYSTEM /F
schtasks /Create /TN "BaccAI_PHD_OnBoot" /TR "C:\Users\Administrator\Desktop\app.py - PHD.bat" /SC ONSTART /RL HIGHEST /RU SYSTEM /F
7. Lessons for the HCU Recovery Period
Sept 7 writing W9 I thought I found the ultimate solution, Sept 23 reality slapped me. This is actually a metaphor for the SEO recovery period:
- Trust no "automatic": watchdog bat looks automatic, but file gets wiped and you're done. Google's algorithm looks automatic, but June 30 put me back to zero.
- Multi-layer redundancy always wins: June 30 I published 47 spam articles, Google demoted me. Sept 23 I added 6-hour forced restart, replaced "automatic" with "forced + multi-layer". Site content should follow the same principle — don't rely on single source (forum / repost / AI), mix PGC + UGC + repost + video.
- Invisible failures are the most lethal: Sept 19 PRO died for 24 hours without me knowing. The final solution includes health check email alerts (in W9 article), combined with W10's 6-hour forced restart, I get notified whenever service dies.