Lab8DATAPROCESSOR/README.md

# Bambuddy InvenTree Sync

Sidecar service for syncing successful Bambuddy print archives into InvenTree stock and tracking filament spools between InvenTree and Bambuddy.

The service runs as a separate Docker container. It does not modify Bambuddy or InvenTree source code. Bambuddy is used as the print-history source, while InvenTree remains the stock-control system of record.

## What It Does

For each successful Bambuddy archive, the service:

- finds or creates an InvenTree `Part`;
- creates one InvenTree `StockItem` for the Bambuddy archive;
- writes `Weight` and `PrintTime` part parameters when those templates exist in InvenTree;
- uploads the Bambuddy archive thumbnail to the InvenTree part image;
- writes an InvenTree part URL back into Bambuddy archive `external_url`;
- stores sync state in SQLite to prevent duplicate stock items.

Failed, stopped, running, or still-printing archives are skipped when `SYNC_SUCCESS_ONLY=true`.

For filament, the service can:

- use InvenTree `StockItem.batch` as the spool identity;
- create/update Bambuddy spool records from InvenTree filament stock;
- move InvenTree spool stock between storage and printer locations from Bambuddy assignments;
- subtract Bambuddy filament usage from the matching InvenTree stock item;
- store usage sync state in SQLite to prevent duplicate subtraction.

## Data Flow

```text
Bambuddy Archives
        |
        | webhook or polling
        v
bambuddy-inventree-sync
        |
        | InvenTree API
        v
Part + StockItem + Parameters + Image
        |
        | Bambuddy API
        v
Archive external_url -> InvenTree Part page
```

Filament flow:

```text
InvenTree filament StockItem.batch
        |
        | creates or updates Bambuddy spool tag_uid
        v
Bambuddy spool + printer assignment
        |
        | stock transfer
        v
InvenTree storage location <-> printer location
        |
        | successful Bambuddy usage history
        v
InvenTree stock/remove subtracts filament grams
```

## InvenTree Mapping

| Bambuddy data | InvenTree target |
| --- | --- |
| archive/model identity | `Part` |
| successful print archive | `StockItem` |
| target category | `Part.category` |
| target storage | `StockItem.location` |
| archive id | `StockItem.batch = bambuddy-<archive_id>` |
| filament weight | `Part` parameter `Weight` |
| print time | `Part` parameter `PrintTime` |
| archive thumbnail | `Part.image` |
| InvenTree part page | Bambuddy `external_url` |

## Filament Tracking

InvenTree remains the source of truth for spool identity and remaining stock.

The key rule is:

```text
InvenTree StockItem.batch == Bambuddy Spool.tag_uid
```

Recommended InvenTree structure for the current setup:

| Purpose | InvenTree ID |
| --- | --- |
| filament part category | `19` |
| filament storage root | `85` |
| loaded-in-printers root | `72` |
| B1 printer stock location | `93` |
| B2 printer stock location | `94` |
| B3 printer stock location | `95` |
| B4 printer stock location | `96` |

The service deliberately starts with `FILAMENT_DRY_RUN=true`. In dry-run mode it reads both systems and reports what it would create, move, or subtract, but it does not write filament changes. Switch to `FILAMENT_DRY_RUN=false` only after `/filament/status` and `/sync/filament?dry_run=true` show the expected mapping.

Filament sync has four independent parts:

- spool catalog sync: InvenTree stock items create/update Bambuddy spools;
- assignment sync: InvenTree printer locations create Bambuddy spool assignments;
- location sync: Bambuddy assignments move InvenTree stock to printer locations; returning unassigned loaded spools to storage is optional;
- usage sync: Bambuddy usage history subtracts grams from the matching InvenTree stock item.

## Duplicate Protection

The service is idempotent:

- each Bambuddy archive ID is stored in `data/sync.sqlite3`;
- each `StockItem` gets batch `bambuddy-<archive_id>`;
- rerunning backfill does not create duplicate stock items;
- repeat prints of the same file/model reuse the same `Part` and create new `StockItem` rows.
- each Bambuddy filament usage ID is stored before it can subtract stock twice.

The `Part` identity key is controlled by:

```env
PART_KEY_FIELDS=content_hash,filename,print_name,name
```

The generated key becomes an InvenTree IPN:

```text
BMB-<first-12-sha1-chars>
```

## Delete Policy

If an archive is deleted in Bambuddy after it was synced, the corresponding InvenTree data is not deleted.

This is intentional. InvenTree is the inventory record, and deleting print history in Bambuddy should not silently remove stock records. The synced `Part`, `StockItem`, parameters, and image remain in InvenTree.

## Requirements

- Docker with Linux containers enabled.
- Network access from the sync container to Bambuddy API and InvenTree API.
- Bambuddy API key.
- InvenTree API token.
- Existing InvenTree part category ID.
- Existing InvenTree stock location ID.
- Optional InvenTree filament category and stock location IDs.
- Optional InvenTree parameter templates: `Weight` and `PrintTime`.

On Windows Server 2022, verify Docker with:

```powershell
docker run --rm hello-world
docker run --rm python:3.12-slim python --version
```

## Docker Deployment

Clone the repository:

```powershell
git clone https://git.tcom.space/tcom/Lab8DATAPROCESSOR.git
cd Lab8DATAPROCESSOR
```

Create the runtime config:

```powershell
Copy-Item .env.example .env
notepad .env
```

Start the service:

```powershell
docker compose up -d --build
```

View logs:

```powershell
docker compose logs --tail=100
```

Stop the service:

```powershell
docker compose down
```

Update to the latest version:

```powershell
git pull
docker compose up -d --build
```

## Configuration

Example `.env`:

```env
BAMBUDDY_BASE_URL=http://192.168.1.5:8000/api/v1
BAMBUDDY_API_KEY=replace-with-bambuddy-api-key

INVENTREE_BASE_URL=http://192.168.0.3:1337
INVENTREE_WEB_URL=
INVENTREE_TOKEN=replace-with-inventree-token
INVENTREE_PART_CATEGORY_ID=26
INVENTREE_STOCK_LOCATION_ID=98

SERVICE_API_TOKEN=change-me
WEBHOOK_SHARED_SECRET=

SYNC_SUCCESS_ONLY=true
SYNC_PART_IMAGES=true
OVERWRITE_PART_IMAGES=false
SYNC_ARCHIVE_EXTERNAL_LINK=true
OVERWRITE_ARCHIVE_EXTERNAL_LINK=false

DEFAULT_STOCK_QUANTITY=1
INVENTREE_STOCK_STATUS=10
PART_IPN_PREFIX=BMB
PART_KEY_FIELDS=content_hash,filename,print_name,name

BACKFILL_PAGE_SIZE=50
POLL_INTERVAL_SECONDS=300
SYNC_ON_STARTUP=false
HTTP_TIMEOUT_SECONDS=30

FILAMENT_TRACKING_ENABLED=false
FILAMENT_DRY_RUN=true
FILAMENT_PART_CATEGORY_ID=19
FILAMENT_STORAGE_LOCATION_ID=85
FILAMENT_LOADED_LOCATION_ID=72
FILAMENT_PRINTER_LOCATION_MAP=B1:93,B2:94,B3:95,B4:96
FILAMENT_BATCH_SOURCE=tag_uid
FILAMENT_SYNC_SPOOLS=true
FILAMENT_SYNC_ASSIGNMENTS=true
FILAMENT_SYNC_LOCATIONS=true
FILAMENT_SYNC_USAGE=true
FILAMENT_RETURN_UNASSIGNED_TO_STORAGE=false
FILAMENT_PRINTER_ID_MAP=
FILAMENT_ASSIGNMENT_DEFAULT_AMS_ID=0
FILAMENT_ASSIGNMENT_START_TRAY_ID=0
FILAMENT_USAGE_LIMIT=200
FILAMENT_USAGE_SUCCESS_STATUSES=success,completed,complete,done
FILAMENT_DEFAULT_MATERIAL=PLA
FILAMENT_DEFAULT_LABEL_WEIGHT=1000
FILAMENT_DEFAULT_CORE_WEIGHT=250

DATA_DIR=/data
```

Do not commit `.env`. It contains API tokens and is ignored by git.

## Important Settings

`BAMBUDDY_BASE_URL`
: Bambuddy API base URL. Use `/api/v1`.

`INVENTREE_BASE_URL`
: InvenTree root URL or `/api` URL. The service normalizes it internally.

`INVENTREE_WEB_URL`
: Browser-facing InvenTree URL for Bambuddy external links. If empty, `INVENTREE_BASE_URL` is used.

`INVENTREE_PART_CATEGORY_ID`
: Existing InvenTree category where auto-created printed parts are placed.

`INVENTREE_STOCK_LOCATION_ID`
: Existing InvenTree stock location where printed stock items are placed.

`SYNC_SUCCESS_ONLY`
: When `true`, only successful/completed prints are imported.

`POLL_INTERVAL_SECONDS`
: Enables automatic periodic backfill. `300` means every 5 minutes. `0` disables polling.

`SYNC_PART_IMAGES`
: Downloads Bambuddy thumbnail and uploads it to InvenTree `Part.image`.

`OVERWRITE_PART_IMAGES`
: When `false`, manually set InvenTree part images are preserved.

`SYNC_ARCHIVE_EXTERNAL_LINK`
: Writes the InvenTree part page URL into Bambuddy `external_url`.

`OVERWRITE_ARCHIVE_EXTERNAL_LINK`
: When `false`, existing non-InvenTree external links in Bambuddy are preserved.

`FILAMENT_TRACKING_ENABLED`
: Enables scheduled and manual filament sync actions.

`FILAMENT_DRY_RUN`
: When `true`, filament endpoints report planned writes but do not create spools, move stock, or subtract stock.

`FILAMENT_PART_CATEGORY_ID`
: InvenTree category containing filament parts.

`FILAMENT_STORAGE_LOCATION_ID`
: InvenTree root location where spare filament spools are stored.

`FILAMENT_LOADED_LOCATION_ID`
: InvenTree root location for spools loaded in printers.

`FILAMENT_PRINTER_LOCATION_MAP`
: Printer prefix to InvenTree location map. Example `B1:93` matches Bambuddy printer names like `B1-X1-CARBON`.

`FILAMENT_BATCH_SOURCE`
: Bambuddy spool field that contains the InvenTree batch code. Default is `tag_uid`.

`FILAMENT_SYNC_SPOOLS`
: Creates/updates Bambuddy spool records from InvenTree stock.

`FILAMENT_SYNC_ASSIGNMENTS`
: Creates Bambuddy assignments from InvenTree stock items currently stored in printer locations.

`FILAMENT_SYNC_LOCATIONS`
: Moves InvenTree stock items between storage and printer locations from Bambuddy assignments.

`FILAMENT_SYNC_USAGE`
: Subtracts successful Bambuddy usage history from InvenTree stock.

`FILAMENT_RETURN_UNASSIGNED_TO_STORAGE`
: When `true`, known Bambuddy spools that are no longer assigned are moved from printer locations back to storage. Keep this `false` until Bambuddy assignments are reliable.

`FILAMENT_PRINTER_ID_MAP`
: Optional explicit printer-name to Bambuddy printer-ID map, for example `B1:5,B2:2,B3:3,B4:4`. If empty, printer IDs are auto-detected from Bambuddy printer names.

`FILAMENT_ASSIGNMENT_DEFAULT_AMS_ID`
: AMS ID used when creating Bambuddy assignments from InvenTree printer locations. Default is `0`.

`FILAMENT_ASSIGNMENT_START_TRAY_ID`
: First tray ID used for each printer when assigning multiple InvenTree-loaded spools. Default is `0`.

## InvenTree IDs

Use numeric IDs for target category and stock location. You can find them from the InvenTree UI URL or API.

Example:

```powershell
curl.exe -H "Authorization: Token YOUR_TOKEN" http://192.168.0.3:1337/api/part/category/26/
curl.exe -H "Authorization: Token YOUR_TOKEN" http://192.168.0.3:1337/api/stock/location/98/
```

Current known example:

```env
INVENTREE_PART_CATEGORY_ID=26
INVENTREE_STOCK_LOCATION_ID=98
```

## Validate Connectivity

After the container starts:

```powershell
curl.exe http://localhost:8088/health
curl.exe -H "X-Service-Token: change-me" http://localhost:8088/validate
curl.exe -H "X-Service-Token: change-me" http://localhost:8088/filament/status
```

Expected result:

- `/health` returns `status: ok`;
- `/validate` confirms Bambuddy, InvenTree category, and InvenTree location are reachable.
- `/filament/status` shows how many InvenTree filament batches match Bambuddy spools.

## Initial Backfill

Test one successful archive first:

```powershell
curl.exe -X POST -H "X-Service-Token: change-me" "http://localhost:8088/sync/backfill?max_archives=1"
```

Run full backfill:

```powershell
curl.exe -X POST -H "X-Service-Token: change-me" http://localhost:8088/sync/backfill
```

Check status:

```powershell
curl.exe -H "X-Service-Token: change-me" http://localhost:8088/sync/status
```

When `SYNC_SUCCESS_ONLY=true`, still-printing or failed archives are skipped and do not consume `max_archives`.

## Automatic Operation

Recommended production mode:

```env
SYNC_SUCCESS_ONLY=true
POLL_INTERVAL_SECONDS=300
SYNC_ON_STARTUP=false
```

This makes the service check Bambuddy every 5 minutes and import all newly completed prints.

To also run filament tracking automatically:

```env
FILAMENT_TRACKING_ENABLED=true
FILAMENT_DRY_RUN=true
FILAMENT_SYNC_SPOOLS=true
FILAMENT_SYNC_ASSIGNMENTS=true
FILAMENT_SYNC_LOCATIONS=true
FILAMENT_SYNC_USAGE=true
FILAMENT_RETURN_UNASSIGNED_TO_STORAGE=false
```

First run in dry-run:

```powershell
curl.exe -X POST -H "X-Service-Token: change-me" "http://localhost:8088/sync/filament?dry_run=true"
```

When the reported actions are correct, change:

```env
FILAMENT_DRY_RUN=false
```

Then rebuild/restart:

```powershell
docker compose up -d --build
```

For near-real-time syncing, also configure a Bambuddy webhook:

```text
http://WINDOWS-SERVER-IP:8088/webhooks/bambuddy
```

Use the `print_complete` event. If `WEBHOOK_SHARED_SECRET` is set, Bambuddy must send:

```text
X-Sync-Secret: your-secret
```

Running both webhook and polling is recommended. Webhook gives fast sync, polling catches missed events after restarts or temporary failures.

## Manual Sync

Sync one archive:

```powershell
curl.exe -X POST -H "X-Service-Token: change-me" http://localhost:8088/sync/archive/11
```

Force re-sync one archive:

```powershell
curl.exe -X POST -H "X-Service-Token: change-me" "http://localhost:8088/sync/archive/11?force=true"
```

Force sync is useful after changing image, parameter, or external link behavior.

Run all filament sync steps manually:

```powershell
curl.exe -X POST -H "X-Service-Token: change-me" http://localhost:8088/sync/filament
```

Run individual filament steps:

```powershell
curl.exe -X POST -H "X-Service-Token: change-me" http://localhost:8088/sync/filament/spools
curl.exe -X POST -H "X-Service-Token: change-me" http://localhost:8088/sync/filament/assignments
curl.exe -X POST -H "X-Service-Token: change-me" http://localhost:8088/sync/filament/locations
curl.exe -X POST -H "X-Service-Token: change-me" http://localhost:8088/sync/filament/usage
```

## API Endpoints

```text
GET  /health
GET  /validate
GET  /sync/status
GET  /filament/status
POST /sync/archive/{archive_id}
POST /sync/backfill
POST /sync/filament
POST /sync/filament/spools
POST /sync/filament/assignments
POST /sync/filament/locations
POST /sync/filament/usage
POST /webhooks/bambuddy
```

Manual sync endpoints require `X-Service-Token` when `SERVICE_API_TOKEN` is set.

## Troubleshooting

View recent logs:

```powershell
docker compose logs --tail=100
```

Rebuild after pulling code changes:

```powershell
git pull
docker compose up -d --build
```

Common issues:

- `401` from service endpoints: missing or wrong `X-Service-Token`.
- `401` from Bambuddy: wrong `BAMBUDDY_API_KEY`.
- `401` from InvenTree: wrong `INVENTREE_TOKEN`.
- InvenTree `description` length errors: update to the latest service version; descriptions are capped at 250 characters.
- No items imported: check if Bambuddy archives are still `printing` or `failed`.
- Duplicate protection prevents repeated imports: use `force=true` only when you want to refresh parameters/images/links for an existing archive.
- Filament sync reports `missing_in_bambuddy`: run `/sync/filament/spools?dry_run=true`, then disable dry-run when the generated spool data is correct.
- Filament usage stays `pending`: the Bambuddy spool is not linked to an InvenTree batch code in `tag_uid`, or the matching InvenTree stock item is missing.
- Location moves do not happen: check `FILAMENT_PRINTER_LOCATION_MAP` and Bambuddy spool assignments.

## Backups

Persistent sync state is stored in:

```text
./data/sync.sqlite3
```

Back up the `data` directory together with `.env` if you move the service to another server.