# IC3 Continuous Innomaint Sync Setup

## Overview
The backend now features:
- ✅ **Continuous background sync worker** - Automatically syncs assets 24/7
- ✅ **Persistent startup** - Backend continues running even if Claude closes
- ✅ **Automatic retry logic** - Failed assets retried with exponential backoff (1s → 2s → 4s)
- ✅ **Batch processing** - Syncs in configurable batch sizes to avoid API rate limits
- ✅ **Sync status endpoints** - Check progress anytime via HTTP API

## Current Status (as of 2026-06-19 15:08 UTC)

```
📊 INNOMAINT API SYNC PROGRESS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total Assets:           2957
✅ Successfully Synced: 381 (12.9%)
⚠️  Need Retry:         166 (5.6%)
⏳ Never Synced:        2426 (82.0%)
⏲️  Sync Interval:      30 seconds
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

## Starting the Backend

### Option 1: Persistent Service (Recommended)
```bash
cd H:\ic3\backend
.\start-backend-service.bat
```
This creates a detached process that survives even if Claude closes.

### Option 2: Direct Startup
```bash
cd H:\ic3\backend
.\ic3-backend.exe
```

## API Endpoints

### Check Sync Status
```bash
curl "http://localhost:9090/api/admin/cmms/sync/status" \
  -H "Authorization: Bearer <token>"
```

**Response:**
```json
{
  "total_assets": 2957,
  "synced_assets": 381,
  "failed_assets": 166,
  "never_synced": 2426,
  "last_sync_at": "2026-06-19T15:08:00Z",
  "last_batch": [...]
}
```

### Sync Batch of Assets (Manual)
```bash
curl -X POST "http://localhost:9090/api/admin/cmms/sync?batch=100&offset=0" \
  -H "Authorization: Bearer <token>"
```

Parameters:
- `batch`: Number of assets per batch (default 100)
- `offset`: Starting position (for pagination)

### Retry Failed Assets
```bash
curl -X POST "http://localhost:9090/api/admin/cmms/sync/retry-failed" \
  -H "Authorization: Bearer <token>"
```

Retries only assets that failed in the last sync attempt.

## How Continuous Sync Works

1. **Automatic Background Worker**
   - Starts when backend launches
   - Runs every 30 seconds (configurable)
   - Processes 100 assets per batch (configurable)

2. **Retry Logic with Exponential Backoff**
   ```
   Failed API calls get retried up to 3 times:
   Attempt 1 → Failed
   Wait 1 second
   Attempt 2 → Failed
   Wait 2 seconds
   Attempt 3 → Failed
   Log to ic3_sync_log for later retry
   ```

3. **Rate Limiting Awareness**
   - 200ms delay between each API request
   - Prevents overwhelming Innomaint API
   - Respects 429 (Too Many Requests) responses

4. **Per-Asset Transactions**
   - Each asset sync is isolated
   - One failure doesn't block others
   - Failed assets logged separately for retry

## Configuration

Edit `cmms_import.go` to adjust:

```go
const (
    innomaintMaxRetries = 3                    // Retry attempts
    innomaintRetryBase  = 1 * time.Second      // Initial backoff
    innomaintAPITimeout = 15 * time.Second     // Per-request timeout
    innomaintRateMS     = 200                  // ms between requests
)
```

Edit `main.go` line 265 to change continuous sync interval:

```go
// Change from:
db.StartContinuousSyncWorker(ctx, 100, 30)
// To:
db.StartContinuousSyncWorker(ctx, 100, 60)  // 60 second interval
```

Parameters: `StartContinuousSyncWorker(ctx, batchSize, intervalSeconds)`

## Understanding Sync Results

### Synced Assets
Assets successfully enriched with Innomaint data including:
- Performance metrics (MTTR, MTBF, availability)
- Work order summaries
- Geolocation data
- Contract information

### Failed Assets
Usually due to:
- API rate limiting (429 errors) - Will retry
- Network timeouts - Will retry
- Asset not found in Innomaint (404) - No retry
- API validation errors - Logged for investigation

### Never Synced
Assets that haven't been processed yet. Continuous worker will eventually reach all.

## Monitoring

### View Backend Logs
```bash
# Check if process is running
tasklist | find /i "ic3-backend"

# Backend logs to stdout/console
# Look for messages like:
# "Continuous sync: 2426 never-synced remaining, syncing next batch..."
# "Continuous sync batch: 10 success, 5 failed, 100 total"
```

### Query Database Directly
```sql
-- See all sync attempts
SELECT * FROM ic3_sync_log 
WHERE sync_type='api_sync' 
ORDER BY synced_at DESC 
LIMIT 100;

-- Count by status
SELECT status, COUNT(*) 
FROM ic3_sync_log 
WHERE sync_type='api_sync' 
GROUP BY status;

-- See batch summaries
SELECT * FROM cmms_sync_log 
WHERE sync_type='api_sync' 
ORDER BY synced_at DESC;
```

## Troubleshooting

### Backend Won't Start
```bash
# Check port 9090 isn't in use
netstat -ano | find "9090"

# Kill existing process if needed
taskkill /F /IM ic3-backend.exe
```

### Sync Stuck
The continuous worker checks every 30 seconds. If stuck:
1. Manually trigger retry: `POST /api/admin/cmms/sync/retry-failed`
2. Check database for errors: `SELECT * FROM ic3_sync_log WHERE status='error'`
3. Review Innomaint API status (might be down or rate-limiting hard)

### Persist Backend After Closing Claude

The backend process is **independent** once started. You can:
1. Close Claude Code completely
2. Backend continues running on localhost:9090
3. Sync continues in background
4. Check status anytime via API

To stop the backend:
```bash
taskkill /F /IM ic3-backend.exe
```

## Next Steps

- Monitor sync progress via status endpoint
- Manually retry failed assets as needed: `POST /api/admin/cmms/sync/retry-failed`
- Once all 2957 assets synced, Innomaint data will be fully integrated
- Use synced data for reports, dashboards, and analytics

## Timeline Estimate

At current rate (~10-15 assets per minute from batch processing):
- Current: 381/2957 synced (12.9%)
- ETA for 100%: ~4-5 hours continuous sync time
- With retries: Total wall-clock time ~6-8 hours

---
**Last Updated:** 2026-06-19 15:08 UTC
**Status:** ✅ Continuous sync active and running