phase 0 & 1 ✅, onto the next logic

AGENTS.md

@@ -11,6 +11,7 @@ This document defines the laws, principles, and rule sets that govern this codeb
 1. **No testing by yourself** — All testing is done by the team.
 2. **No assumptions about code or domain logic** — Always confirm and be sure before making changes.
 3. **No running scripts** — Do not run build, dev, test, or migrate scripts unless explicitly approved.
+4. **No touching migration files** — Do not mess with the `migrations` sql dir, as those are generated manually via drizzle orm
 
 More rules are only to be added by the human, in case such a suggestion becomes viable.
 

README.md

@@ -1,3 +1,199 @@
 # Illusory MAPP
 
-An internal automation platform built collaboratively. The premise is simple: aggregate data from each member's personal devices — mobile and otherwise — into a single centralized hub, then execute complex group-based automations on top of that collective data. The end goal is a shared automation center where the group's combined device data drives coordinated, programmable workflows.
+Internal automation platform for aggregating device data into one backend, then running coordinated automations on top of that data.
+
+## Current Goal (Server-Side Stage 1)
+
+Build the server-side ingestion and admin visibility for:
+
+1. Device registration
+2. Device heartbeat/ping tracking
+3. SMS ingestion (ongoing)
+4. Media metadata ingestion (initial sync)
+5. Admin UI to inspect devices, SMS, and media
+
+Mobile app implementation is out of scope for now.
+
+## Scope and Non-Goals
+
+### In scope now
+
+- Processor endpoints that receive mobile data
+- Domain logic + persistence for devices, SMS, media sync metadata
+- Admin UI read views with polling for SMS (every 5 seconds)
+- Observability for ingestion and read flows (logs + traces)
+
+### Not in scope now
+
+- Mobile app UX/screens or client implementation
+- Automation execution workflows
+- Advanced media processing pipelines
+
+## Delivery Plan (Actionable Tasks)
+
+Use this as the step-by-step implementation checklist.
+
+### Phase 0: Align Critical Decisions (before coding)
+
+- [x] Confirm device identity strategy (`deviceId` from mobile vs server-generated key).
+- [x] Confirm admin ownership rule for newly registered devices (single admin user mapping).
+- [x] Confirm SMS dedup strategy (client message id, hash, or `(deviceId + timestamp + sender + body)`).
+- [x] Confirm media sync contract: metadata-only now vs metadata + upload references.
+- [x] Confirm minimal auth for processor endpoints (shared token or signed header) for Stage 1.
+
+Decisions captured:
+
+- Keep two identifiers per device:
+    - `id` (server-generated primary key)
+    - `externalDeviceId` (sent by mobile)
+- All devices are owned by the single admin user in Stage 1.
+- SMS dedup uses:
+    - optional client message id when available
+    - fallback deterministic hash from `(deviceId + timestamp + sender + body)`.
+- Media sync is metadata plus upload reference (`fileId`) to the file domain/R2 object.
+- Stage 1 endpoint auth:
+    - register is open in trusted network
+    - ping/sync endpoints must include device id header; request is allowed only if device exists.
+
+### Phase 1: Data Model and DB Migrations
+
+Target: `packages/db/schema/*` + new migration files.
+
+- [ ] Add `mobile_device` table:
+    - Fields: `id`, `externalDeviceId`, `name`, `manufacturer`, `model`, `androidVersion`, `ownerUserId`, `lastPingAt`, `createdAt`, `updatedAt`.
+- [ ] Add `mobile_sms` table:
+    - Fields: `id`, `deviceId`, `externalMessageId?`, `sender`, `recipient?`, `body`, `sentAt`, `receivedAt?`, `rawPayload?`, `createdAt`.
+- [ ] Add `mobile_media_asset` table:
+    - Fields: `id`, `deviceId`, `externalMediaId?`, `mimeType`, `filename?`, `capturedAt?`, `sizeBytes?`, `hash?`, `metadata`, `createdAt`.
+- [ ] Add indexes for common reads:
+    - `mobile_device.lastPingAt`
+    - `mobile_sms.deviceId + sentAt desc`
+    - `mobile_media_asset.deviceId + createdAt desc`
+- [ ] Export schema from `packages/db/schema/index.ts`.
+
+Definition of done:
+
+- [ ] Tables and indexes exist in schema + migration files.
+- [ ] Naming matches existing conventions.
+
+### Phase 2: Logic Domain (`@pkg/logic`)
+
+Target: `packages/logic/domains/mobile/*` (new domain).
+
+- [ ] Create `data.ts` with Valibot schemas and inferred types for:
+    - register device payload
+    - ping payload
+    - sms sync payload
+    - media sync payload
+    - admin query filters/pagination
+- [ ] Create `errors.ts` with domain error constructors using `getError`.
+- [ ] Create `repository.ts` with ResultAsync operations for:
+    - upsert device
+    - update last ping
+    - bulk insert sms (idempotent)
+    - bulk insert media metadata (idempotent)
+    - list devices
+    - get device detail
+    - list sms by device (paginated, latest first)
+    - list media by device (paginated, latest first)
+    - delete single media asset
+    - delete device (removed all child sms, and media assets, and by extensionn the associated files in r2+db as well)
+- [ ] Create `controller.ts` as use-case layer.
+- [ ] Wrap repository/controller operations with `traceResultAsync` and include `fctx`.
+
+Definition of done:
+
+- [ ] Processor and main app can consume this domain without direct DB usage.
+- [ ] No ad-hoc thrown errors in domain flow; Result pattern is preserved.
+
+### Phase 3: Processor Ingestion API (`apps/processor`)
+
+Target: `apps/processor/src/domains/mobile/router.ts`.
+
+- [ ] Replace stub endpoints with full handlers:
+    - `POST /register`
+    - `PUT /ping`
+    - `PUT /sms/sync`
+    - `PUT /media/sync`
+- [ ] Validate request body using schemas from `@pkg/logic/domains/mobile/data`.
+- [ ] Build `FlowExecCtx` per request (flow id always; user/session when available).
+- [ ] Call mobile controller methods; return `{ data, error }` response shape.
+- [ ] Add basic endpoint protection agreed in Phase 0.
+- [ ] Add request-level structured logging for success/failure.
+
+Definition of done:
+
+- [ ] Endpoints persist data into new tables.
+- [ ] Endpoint failures return normalized domain errors.
+
+### Phase 4: Admin UI in `apps/main`
+
+Target:
+
+- `apps/main/src/lib/domains/mobile/*` (new)
+- `apps/main/src/routes/(main)/devices/*` (new)
+
+- [ ] Add remote functions:
+    - `getDevicesSQ`
+    - `getDeviceDetailSQ`
+    - `getDeviceSmsSQ`
+    - `getDeviceMediaSQ`
+- [ ] Add VM (`devices.vm.svelte.ts`) that:
+    - fetches devices list
+    - fetches selected device detail
+    - polls SMS every 5 seconds while device view is open
+    - handles loading/error states
+- [ ] Add UI pages/components:
+    - `/devices` list with device identity + last ping
+    - `/devices/[deviceId]` detail with tabs:
+        - Device info
+        - SMS feed
+        - Media assets list
+- [ ] Add sidebar/navigation entry for Devices.
+
+Definition of done:
+
+- [ ] Admin can browse devices and open each device detail.
+- [ ] SMS view refreshes every 5 seconds and shows new data.
+
+### Phase 5: Observability Stage 1
+
+Targets:
+
+- `packages/logic/core/observability.ts` (use existing helpers)
+- Processor/mobile domain handlers and repository/controller paths
+
+- [ ] Add span names for key flows:
+    - `mobile.register`
+    - `mobile.ping`
+    - `mobile.sms.sync`
+    - `mobile.media.sync`
+    - `mobile.devices.list`
+    - `mobile.device.detail`
+- [ ] Add structured domain events with device id, counts, durations.
+- [ ] Ensure errors include `flowId` consistently.
+
+Definition of done:
+
+- [ ] Can trace one request from processor endpoint to DB operation via shared `flowId`.
+
+### Phase 6: Handoff Readiness (Team Test Phase)
+
+- [ ] Prepare endpoint payload examples for mobile team.
+- [ ] Document pagination/sorting contract for admin UI queries.
+- [ ] Document dedup behavior for SMS/media ingestion.
+- [ ] Provide a short operator checklist:
+    - register device
+    - verify ping updates
+    - verify sms appears in admin
+    - verify media appears in admin
+
+## Suggested Build Order
+
+1. Phase 0
+2. Phase 1
+3. Phase 2
+4. Phase 3
+5. Phase 4
+6. Phase 5
+7. Phase 6

apps/processor/src/domains/mobile/router.ts

@@ -1,5 +1,15 @@
 import { Hono } from "hono";
 
-const mobileRouter = new Hono().get("/", async (c) => {
-    return c.json({ message: "" });
-});
+export const mobileRouter = new Hono()
+    .put("/ping", async (c) => {
+        return c.json({ data: "" });
+    })
+    .put("/sms/sync", async (c) => {
+        return c.json({ data: "" });
+    })
+    .put("/media/sync", async (c) => {
+        return c.json({ data: "" });
+    })
+    .post("/register", async (c) => {
+        return c.json({ data: "" });
+    });

apps/processor/src/index.ts

@@ -1,12 +1,19 @@
+import { mobileRouter } from "./domains/mobile/router.js";
 import { serve } from "@hono/node-server";
 import { Hono } from "hono";
 
 const app = new Hono();
 
-app.get("/", (c) => {
-    return c.text("Hello Hono!");
+app.get("/health", (c) => {
+    return c.json({ ok: true });
 });
 
+app.get("/ping", (c) => {
+    return c.text("pong");
+});
+
+app.route("/api/v1/mobile", mobileRouter);
+
 serve(
     {
         fetch: app.fetch,

packages/db/migrations/0002_massive_captain_britain.sql

@@ -0,0 +1,56 @@
+CREATE TABLE "mobile_device" (
+	"id" serial PRIMARY KEY NOT NULL,
+	"external_device_id" text NOT NULL,
+	"name" text NOT NULL,
+	"manufacturer" text NOT NULL,
+	"model" text NOT NULL,
+	"android_version" text NOT NULL,
+	"owner_user_id" text NOT NULL,
+	"last_ping_at" timestamp,
+	"created_at" timestamp NOT NULL,
+	"updated_at" timestamp NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "mobile_media_asset" (
+	"id" serial PRIMARY KEY NOT NULL,
+	"device_id" integer NOT NULL,
+	"external_media_id" text,
+	"file_id" text NOT NULL,
+	"mime_type" text NOT NULL,
+	"filename" text,
+	"captured_at" timestamp,
+	"size_bytes" integer,
+	"hash" text,
+	"metadata" json,
+	"created_at" timestamp NOT NULL,
+	"updated_at" timestamp NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "mobile_sms" (
+	"id" serial PRIMARY KEY NOT NULL,
+	"device_id" integer NOT NULL,
+	"external_message_id" text,
+	"sender" text NOT NULL,
+	"recipient" text,
+	"body" text NOT NULL,
+	"sent_at" timestamp NOT NULL,
+	"received_at" timestamp,
+	"dedup_hash" text NOT NULL,
+	"raw_payload" json,
+	"created_at" timestamp NOT NULL,
+	"updated_at" timestamp NOT NULL
+);
+--> statement-breakpoint
+ALTER TABLE "mobile_device" ADD CONSTRAINT "mobile_device_owner_user_id_user_id_fk" FOREIGN KEY ("owner_user_id") REFERENCES "public"."user"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "mobile_media_asset" ADD CONSTRAINT "mobile_media_asset_device_id_mobile_device_id_fk" FOREIGN KEY ("device_id") REFERENCES "public"."mobile_device"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "mobile_media_asset" ADD CONSTRAINT "mobile_media_asset_file_id_file_id_fk" FOREIGN KEY ("file_id") REFERENCES "public"."file"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "mobile_sms" ADD CONSTRAINT "mobile_sms_device_id_mobile_device_id_fk" FOREIGN KEY ("device_id") REFERENCES "public"."mobile_device"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+CREATE UNIQUE INDEX "mobile_device_external_device_id_uq" ON "mobile_device" USING btree ("external_device_id");--> statement-breakpoint
+CREATE INDEX "mobile_device_owner_user_id_idx" ON "mobile_device" USING btree ("owner_user_id");--> statement-breakpoint
+CREATE INDEX "mobile_device_last_ping_at_idx" ON "mobile_device" USING btree ("last_ping_at");--> statement-breakpoint
+CREATE INDEX "mobile_media_asset_device_created_at_idx" ON "mobile_media_asset" USING btree ("device_id","created_at");--> statement-breakpoint
+CREATE UNIQUE INDEX "mobile_media_asset_device_external_media_uq" ON "mobile_media_asset" USING btree ("device_id","external_media_id");--> statement-breakpoint
+CREATE UNIQUE INDEX "mobile_media_asset_file_id_uq" ON "mobile_media_asset" USING btree ("file_id");--> statement-breakpoint
+CREATE INDEX "mobile_sms_device_sent_at_idx" ON "mobile_sms" USING btree ("device_id","sent_at");--> statement-breakpoint
+CREATE UNIQUE INDEX "mobile_sms_device_dedup_hash_uq" ON "mobile_sms" USING btree ("device_id","dedup_hash");--> statement-breakpoint
+CREATE UNIQUE INDEX "mobile_sms_device_external_msg_uq" ON "mobile_sms" USING btree ("device_id","external_message_id");

packages/db/migrations/meta/_journal.json

@@ -15,6 +15,13 @@
       "when": 1772335785371,
       "tag": "0001_silly_venus",
       "breakpoints": true
+    },
+    {
+      "idx": 2,
+      "version": "7",
+      "when": 1772338633827,
+      "tag": "0002_massive_captain_britain",
+      "breakpoints": true
     }
   ]
 }

packages/db/schema/file.schema.ts

@@ -11,7 +11,6 @@ import {
 import { user } from "./better.auth.schema";
 import { relations } from "drizzle-orm";
 
-// Add this to your existing schema file
 export const file = pgTable("file", {
     id: text("id").primaryKey(), // UUID
 

packages/db/schema/index.ts

@@ -2,3 +2,4 @@ export * from "./auth.schema";
 export * from "./better.auth.schema";
 export * from "./file.schema";
 export * from "./general.schema";
+export * from "./mobile.device.schema";

packages/db/schema/mobile.device.schema.ts

@@ -0,0 +1,145 @@
+import {
+    index,
+    integer,
+    json,
+    pgTable,
+    serial,
+    text,
+    timestamp,
+    uniqueIndex,
+} from "drizzle-orm/pg-core";
+import { user } from "./better.auth.schema";
+import { relations } from "drizzle-orm";
+import { file } from "./file.schema";
+
+export const mobileDevice = pgTable(
+    "mobile_device",
+    {
+        id: serial("id").primaryKey(),
+        externalDeviceId: text("external_device_id").notNull(),
+        name: text("name").notNull(),
+        manufacturer: text("manufacturer").notNull(),
+        model: text("model").notNull(),
+        androidVersion: text("android_version").notNull(),
+        ownerUserId: text("owner_user_id")
+            .notNull()
+            .references(() => user.id, { onDelete: "cascade" }),
+        lastPingAt: timestamp("last_ping_at"),
+
+        createdAt: timestamp("created_at").notNull(),
+        updatedAt: timestamp("updated_at").notNull(),
+    },
+    (table) => ({
+        externalDeviceUniqueIdx: uniqueIndex(
+            "mobile_device_external_device_id_uq",
+        ).on(table.externalDeviceId),
+        ownerUserIdx: index("mobile_device_owner_user_id_idx").on(
+            table.ownerUserId,
+        ),
+        lastPingIdx: index("mobile_device_last_ping_at_idx").on(
+            table.lastPingAt,
+        ),
+    }),
+);
+
+export const mobileSMS = pgTable(
+    "mobile_sms",
+    {
+        id: serial("id").primaryKey(),
+        deviceId: integer("device_id")
+            .notNull()
+            .references(() => mobileDevice.id, { onDelete: "cascade" }),
+        externalMessageId: text("external_message_id"),
+        sender: text("sender").notNull(),
+        recipient: text("recipient"),
+        body: text("body").notNull(),
+        sentAt: timestamp("sent_at").notNull(),
+        receivedAt: timestamp("received_at"),
+        dedupHash: text("dedup_hash").notNull(),
+        rawPayload: json("raw_payload").$type<Record<string, any>>(),
+
+        createdAt: timestamp("created_at").notNull(),
+        updatedAt: timestamp("updated_at").notNull(),
+    },
+    (table) => ({
+        deviceSentIdx: index("mobile_sms_device_sent_at_idx").on(
+            table.deviceId,
+            table.sentAt,
+        ),
+        dedupHashUniqueIdx: uniqueIndex("mobile_sms_device_dedup_hash_uq").on(
+            table.deviceId,
+            table.dedupHash,
+        ),
+        externalMessageUniqueIdx: uniqueIndex(
+            "mobile_sms_device_external_msg_uq",
+        ).on(table.deviceId, table.externalMessageId),
+    }),
+);
+
+export const mobileMediaAsset = pgTable(
+    "mobile_media_asset",
+    {
+        id: serial("id").primaryKey(),
+        deviceId: integer("device_id")
+            .notNull()
+            .references(() => mobileDevice.id, { onDelete: "cascade" }),
+        externalMediaId: text("external_media_id"),
+        fileId: text("file_id")
+            .notNull()
+            .references(() => file.id, { onDelete: "cascade" }),
+        mimeType: text("mime_type").notNull(),
+        filename: text("filename"),
+        capturedAt: timestamp("captured_at"),
+        sizeBytes: integer("size_bytes"),
+        hash: text("hash"),
+        metadata: json("metadata").$type<Record<string, any>>(),
+
+        createdAt: timestamp("created_at").notNull(),
+        updatedAt: timestamp("updated_at").notNull(),
+    },
+    (table) => ({
+        deviceCreatedIdx: index("mobile_media_asset_device_created_at_idx").on(
+            table.deviceId,
+            table.createdAt,
+        ),
+        externalMediaUniqueIdx: uniqueIndex(
+            "mobile_media_asset_device_external_media_uq",
+        ).on(table.deviceId, table.externalMediaId),
+        fileUniqueIdx: uniqueIndex("mobile_media_asset_file_id_uq").on(
+            table.fileId,
+        ),
+    }),
+);
+
+export const mobileDeviceRelations = relations(
+    mobileDevice,
+    ({ one, many }) => ({
+        owner: one(user, {
+            fields: [mobileDevice.ownerUserId],
+            references: [user.id],
+        }),
+        sms: many(mobileSMS),
+        mediaAssets: many(mobileMediaAsset),
+    }),
+);
+
+export const mobileSMSRelations = relations(mobileSMS, ({ one }) => ({
+    device: one(mobileDevice, {
+        fields: [mobileSMS.deviceId],
+        references: [mobileDevice.id],
+    }),
+}));
+
+export const mobileMediaAssetRelations = relations(
+    mobileMediaAsset,
+    ({ one }) => ({
+        device: one(mobileDevice, {
+            fields: [mobileMediaAsset.deviceId],
+            references: [mobileDevice.id],
+        }),
+        file: one(file, {
+            fields: [mobileMediaAsset.fileId],
+            references: [file.id],
+        }),
+    }),
+);