Time On Tasks & Operations Runbook

Process 2: Backup and Recovery

Both LaneAward environments are protected by the 🚀 PWA BDR service — a shared macOS menu bar LaunchAgent on the development Mac. Nothing needs to be installed on the VM. Look for the 🚀 icon in the top menu bar to open the separate Lensboard and LaneAward dropdowns, trigger manual runs, adjust schedules, or mark jobs as Include or Skip.

sudo cp /var/lib/laneaward-staging/workforce.db /var/lib/laneaward-staging/workforce-$(date +%F-%H%M%S).db

Process 3: Staging Status Check

Use the VM status helper when you want one quick read on service health, HTTP routes, and database row counts.

vm_workforce_status.sh

SERVICE_NAME=laneaward-workforce-api-staging.service DB_DIR=/var/lib/laneaward-staging DIRECT_HEALTH_URL=http://127.0.0.1:9193/api/health NGINX_HEALTH_URL=https://staging.timeontasks.laneaward.com/api/health ROOT_URL=https://staging.timeontasks.laneaward.com/ CONSOLE_URL=https://staging.console.laneaward.com/ LEGACY_WORKFORCE_URL=https://staging.timeontasks.laneaward.com/workforce/ /opt/laneaward-staging/workforce_app/deploy/vm_workforce_status.sh

Process 8: Reliability Verification For Task Writes

Phase 1 durability hardening is now part of the backend and should be preserved whenever the application programming interface, or API, is updated. The Time On Tasks API now opens SQLite in Write-Ahead Logging (WAL) mode, waits up to 10 seconds for short lock contention, uses synchronous = FULL for safer commits, and wraps each mutating route in a short BEGIN IMMEDIATE write transaction.

Iteration test checklist:

• Open two or more contributor-capable sessions on staging.
• Rapidly add, start, pause, complete, and cancel tasks from separate browsers or tablets.
• Confirm each UI action returns to a stable task state without duplicate sessions or duplicate task inserts.
• If a write is delayed by contention, confirm the API reports a retryable busy condition instead of a generic server error.
• After the test pass, verify the affected task lists and completed records still match the contributors who performed the actions.

Process 9: Service Worker — Deploy Model And Cache Invalidation

Time On Tasks includes a service worker at timeontasks/sw.js that improves load speed on shared tablets by caching static assets locally after the first visit. Understanding the cache strategy is important before deploying frontend changes.

Process 10: Concurrent Stress Test

Run this test against staging to confirm that the Phase 1 reliability improvements (WAL, busy_timeout, synchronous = FULL, BEGIN IMMEDIATE) hold up under the expected concurrent load of up to 20 simultaneous tablet operators on the shop floor. This test was completed before go-live and passed cleanly. The procedure is preserved here as a reference for future validation runs (e.g. after significant backend changes or scale increases).

The test script is at workforce_app/backend/stress_test_concurrent.py and runs from your local Mac. It seeds its own test fixtures into the staging database via SSH, runs the load, then cleans up after itself.

Prerequisites

• SSH key access to ubuntu@3.130.69.109 with no passphrase prompt (BatchMode).
• Staging server running and reachable at https://staging.timeontasks.laneaward.com.
• Python 3.10 or later on your local Mac — no third-party packages required.
• Run from the repo root: cd ~/Project-Code/laneaward/repo

Step 1 — Verify SSH access before starting

The test seeds and cleans up via SSH. Confirm it works first:

ssh -i ~/.ssh/lane_webserver.pem ubuntu@3.130.69.109 echo "SSH OK"

You should see SSH OK with no password prompt. If you see Permission denied (publickey), the key path is wrong or the key is not present — check that ~/.ssh/lane_webserver.pem exists on your Mac.

Step 2 — Run the standard realistic test

This is the test to run before go-live. It simulates 20 human-paced operators for 2 task cycles each. Expected wall time is roughly 20–40 seconds.

python3 workforce_app/backend/stress_test_concurrent.py

The script will print its progress as it seeds, runs, and cleans up:

[seed]  Seeding 20 test user(s) on ubuntu@3.130.69.109 ...
[check] Verifying https://staging.timeontasks.laneaward.com/api/health ...
[run]   Releasing 20 threads simultaneously ...
[cleanup] Removing stress-test fixtures from remote DB ...

Step 3 — Read the report

At the end of the run, the script prints a report. A clean pass looks like this:

====================================================================
  LANEAWARD TIME-ON-TASKS  —  CONCURRENT STRESS TEST
====================================================================
  Mode:             REALISTIC (human-paced 1.5–4.0 s)
  Target:           https://staging.timeontasks.laneaward.com
  Concurrent users: 20
  Iterations/user:  2
  Total ops:        280
  Wall time:        31.4s

  Outcomes:
    Successes:        280  (100%)
    Hard failures:      0  (non-retryable errors or timeouts)
    Retryable busy:     0  (SQLite busy-wait — server queued OK)

  End-to-end latency (successful ops only):
    p50:    210 ms
    p75:    310 ms
    p95:    480 ms
    max:    740 ms

  VERDICT
  ------------------------------------------------------------------
  PASS  All operations completed cleanly under concurrent load.
        WAL + BEGIN IMMEDIATE handled 20-user contention with no busy errors.

  Latency looks good for tablet UX (p95 = 480 ms).

Step 4 — Optional: run the burst ceiling test

The burst test fires all writes near-simultaneously with no human delay. This scenario cannot occur with real operators, but it validates the SQLite busy-timeout safety net as an absolute ceiling. Run it after the realistic test passes.

python3 workforce_app/backend/stress_test_concurrent.py --burst

Expect more retryable busy responses in this mode — that is normal and expected. The important thing is still zero hard failures.

Adjusting the test

If the test fails

bash scripts/cleanup_stress_test_staging.sh

Process 11: Server Access And Security Group Management

The server is protected by two layers: an AWS Security Group that restricts SSH (port 22) to authorized IP addresses, and a Twingate connector that allows SSH from any location through the Twingate client. Use this process to connect to the server and to update security group rules when IP addresses change. This process applies to both the staging and production environments since both run on the same AWS server.

Option A — SSH via Twingate (from any location)

Use this method when connecting from home or any location not on an authorized static IP. The Twingate client must be running and connected before opening SSH.

Step 1 — Open the Twingate client on your Mac and confirm it shows Connected

The Twingate icon lives in the Mac menu bar. Click it and verify the connection status is active.

Step 2 — SSH using the server private IP

[Mac-local]
ssh -i ~/.ssh/lane_webserver.pem ubuntu@172.31.7.224

Option B — SSH directly from an authorized office IP

Use this method when connecting from the office on either the fiber or cable connection. Twingate does not need to be running.

Step 1 — SSH using the server public IP

[Mac-local]
ssh -i ~/.ssh/lane_webserver.pem ubuntu@3.130.69.109

Check Twingate Connector Status

Run this on the server to confirm the connector service is running. A healthy connector shows State: Online in the log output.

[VM]
sudo systemctl status twingate-connector --no-pager

Update Security Group — Replace a Changed IP

Run these steps when an authorized IP address changes. Requires the AWS CLI configured on the development Mac with IAM user donald.

Step 1 — Verify current rules before making changes

[Mac-local]
aws ec2 describe-security-groups --group-ids sg-0cc9719fa0e029c40 --query "SecurityGroups[0].IpPermissions" --output json --no-cli-pager

Step 2 — Remove the old IP

[Mac-local]
aws ec2 revoke-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr OLD.IP.ADDRESS/32

Step 3 — Add the new IP

[Mac-local]
aws ec2 authorize-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr NEW.IP.ADDRESS/32

Step 4 — Verify the updated rules

[Mac-local]
aws ec2 describe-security-groups --group-ids sg-0cc9719fa0e029c40 --query "SecurityGroups[0].IpPermissions" --output json --no-cli-pager

Remove the Home Lab IP (Pending Task)

Once Twingate is confirmed as the primary home access method, run this to remove the dynamic home lab IP. Do not run this until Twingate SSH has been verified working from the home location.

[Mac-local]
aws ec2 revoke-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr 72.208.129.218/32

Salary Labor Cost — Calculation Methodology

Salaried contributors log time through the Time On Tasks app exactly like hourly workers. The Order Profitability report converts their annual salary to an effective hourly rate using the U.S. Bureau of Labor Statistics standard:

Staging Notes

• The activity reset (Process 4) intentionally preserves all user accounts. Staging contains a mix of real contributor accounts and developer test accounts — both are valid and should be retained.
• Both Operations Console reports (Contributor Task Activity and Order Profitability) are fully implemented and live on staging and production.
• Network and concurrent stress testing completed — Phase 1 database hardening is sufficient at current scale.