citrus-cms/resources/views/guide/QUEUE_MANAGEMENT_GUIDE.md

# Queue Management Guide - viksa2

## Overview

The viksa2 application uses separate queue channels to ensure that different types of jobs don't block each other. This architecture allows heavy, long-running jobs and quick, short-running jobs to execute in parallel without interference.

## Queue Architecture

### Queue Channels

We have three separate queue channels configured:

1. **`default`** - General purpose queue for miscellaneous jobs
2. **`register-creator`** - Dedicated queue for RegisterCreatorJob (heavy, long-running operations)
3. **`save-trigger`** - Dedicated queue for ExecuteSaveTriggerJob (quick trigger execution)

### Configuration

All queue configurations are defined in `config/queue.php`:

```php
'register-creator' => [
    'driver' => 'database',
    'table' => 'jobs',
    'queue' => 'register-creator',
    'retry_after' => 3600, // 1 hour timeout for long-running operations
    'after_commit' => false,
],

'save-trigger' => [
    'driver' => 'database',
    'table' => 'jobs',
    'queue' => 'save-trigger',
    'retry_after' => 90, // 90 seconds for quick trigger execution
    'after_commit' => false,
],
```

## Job Configuration

### RegisterCreatorJob

- **Queue**: `register-creator`
- **Timeout**: 3600 seconds (1 hour)
- **Retry Attempts**: 3
- **Workers**: 2 processes

This job handles heavy PDF generation and Excel processing operations. It's configured to run for extended periods without timing out.

**Location**: `app/Jobs/RegisterCreatorJob.php`

```php
public $queue = 'register-creator';
public int $tries = 3;
public int $timeout = 3600;
```

### ExecuteSaveTriggerJob

- **Queue**: `save-trigger`
- **Timeout**: 300 seconds (5 minutes)
- **Retry Attempts**: 3
- **Workers**: 4 processes

This job handles quick database trigger executions. Multiple workers ensure fast processing of trigger operations.

**Location**: `app/Jobs/ExecuteSaveTriggerJob.php`

```php
public $queue = 'save-trigger';
```

## Supervisor Configuration

Supervisor manages the queue workers as persistent background processes. The configuration is located at:
`/etc/supervisor/conf.d/laravel-worker.conf`

### Worker Processes

#### Default Worker
```ini
[program:viksa2-worker]
command=php /var/www/html/viksa2/artisan queue:work --queue=default
numprocs=1
```

#### Register Creator Worker
```ini
[program:viksa2-register-creator]
command=php /var/www/html/viksa2/artisan queue:work --queue=register-creator
numprocs=2
timeout=3600
```
- Runs 2 parallel processes
- Designed for long-running operations

#### Save Trigger Worker
```ini
[program:viksa2-save-trigger]
command=php /var/www/html/viksa2/artisan queue:work --queue=save-trigger
numprocs=4
timeout=300
```
- Runs 4 parallel processes
- Designed for quick execution with high throughput

## Management Commands

### Supervisor Commands

```bash
# Reload supervisor configuration
sudo supervisorctl reread

# Update supervisor with new configuration
sudo supervisorctl update

# Start all viksa2 workers
sudo supervisorctl start viksa2-worker:*
sudo supervisorctl start viksa2-register-creator:*
sudo supervisorctl start viksa2-save-trigger:*

# Stop all viksa2 workers
sudo supervisorctl stop viksa2-worker:*
sudo supervisorctl stop viksa2-register-creator:*
sudo supervisorctl stop viksa2-save-trigger:*

# Restart all viksa2 workers
sudo supervisorctl restart viksa2-worker:*
sudo supervisorctl restart viksa2-register-creator:*
sudo supervisorctl restart viksa2-save-trigger:*

# Check status
sudo supervisorctl status

# Restart individual worker group
sudo supervisorctl restart viksa2-register-creator:*
```

### Queue Monitoring

```bash
# Monitor queue status
php artisan queue:monitor register-creator,save-trigger,default --max=100

# List failed jobs
php artisan queue:failed

# Retry all failed jobs
php artisan queue:retry all

# Retry specific failed job
php artisan queue:retry {job-id}

# Clear all failed jobs
php artisan queue:flush

# Check queue statistics
php artisan queue:work --once --queue=register-creator

# Process only one job (useful for testing)
php artisan queue:work --once --queue=save-trigger
```

### Log Monitoring

```bash
# Watch register creator worker logs
tail -f /var/www/html/viksa2/storage/logs/register-creator-worker.log

# Watch save trigger worker logs
tail -f /var/www/html/viksa2/storage/logs/save-trigger-worker.log

# Watch general worker logs
tail -f /var/www/html/viksa2/storage/logs/worker.log

# Watch Laravel application logs
tail -f /var/www/html/viksa2/storage/logs/laravel.log

# Watch all worker logs simultaneously
tail -f /var/www/html/viksa2/storage/logs/*-worker.log
```

## Deployment Workflow

When deploying changes that affect jobs:

```bash
# 1. Pull latest code
cd /var/www/html/viksa2
git pull

# 2. Update dependencies (if needed)
composer install --no-dev --optimize-autoloader

# 3. Clear caches
php artisan config:clear
php artisan cache:clear
php artisan queue:restart

# 4. Restart supervisor workers
sudo supervisorctl restart viksa2-worker:*
sudo supervisorctl restart viksa2-register-creator:*
sudo supervisorctl restart viksa2-save-trigger:*

# 5. Verify workers are running
sudo supervisorctl status | grep viksa2
```

## Troubleshooting

### Workers Not Processing Jobs

```bash
# Check if workers are running
sudo supervisorctl status | grep viksa2

# Check if jobs are queued in database
mysql -e "SELECT * FROM viksa2_db.jobs ORDER BY id DESC LIMIT 10;"

# Check worker logs for errors
tail -100 /var/www/html/viksa2/storage/logs/register-creator-worker.log
tail -100 /var/www/html/viksa2/storage/logs/save-trigger-worker.log
```

### High Memory Usage

If workers consume too much memory:

```bash
# Add memory limit to worker command in supervisor config
command=php -d memory_limit=2048M /var/www/html/viksa2/artisan queue:work

# Or set max-time to restart workers periodically
command=php /var/www/html/viksa2/artisan queue:work --max-time=3600
```

### Job Timeouts

If jobs are timing out:

1. Increase `timeout` in job class
2. Increase `retry_after` in queue config
3. Increase `stopwaitsecs` in supervisor config

### Failed Jobs

```bash
# View failed job details
php artisan queue:failed

# Retry failed jobs
php artisan queue:retry all

# Delete failed jobs older than 48 hours
php artisan queue:prune-failed --hours=48
```

## Performance Tuning

### Adjust Worker Count

Based on server resources:

- **Low traffic**: 1 register-creator, 2 save-trigger
- **Medium traffic**: 2 register-creator, 4 save-trigger (current)
- **High traffic**: 4 register-creator, 8 save-trigger

Edit `/etc/supervisor/conf.d/laravel-worker.conf` and change `numprocs` value.

### Sleep Time Optimization

- `--sleep=1` for high-priority queues (save-trigger)
- `--sleep=3` for normal queues (default, register-creator)
- Lower values = more responsive, higher CPU usage

### Connection Pooling

For database driver, ensure sufficient database connections:

```
Max Connections = (Workers × Processes) + Application Connections + Buffer
```

Example: (3 worker groups × 7 total processes) + 20 app connections + 10 buffer = 51 connections

## Best Practices

1. **Always use `queue:restart` after code changes** - Workers cache the application state
2. **Monitor failed jobs regularly** - Set up alerts for failed job threshold
3. **Use `--max-time` parameter** - Prevents memory leaks by restarting workers periodically
4. **Separate heavy and light jobs** - Use dedicated queues for different job types
5. **Log everything** - Use Laravel's logging extensively in job classes
6. **Test with `--once` flag** - Test job execution before deploying to production
7. **Set realistic timeouts** - Configure timeouts based on actual job execution times
8. **Use job tags** - Makes debugging easier with Telescope or Horizon

## Queue Priority

If you need to process multiple queues with priority ordering:

```bash
# Higher priority queues come first
php artisan queue:work --queue=save-trigger,register-creator,default
```

This ensures `save-trigger` jobs are processed before `register-creator` jobs.

## Monitoring Tools

### Laravel Horizon (Optional)

For advanced queue monitoring, consider installing Laravel Horizon:

```bash
composer require laravel/horizon
php artisan horizon:install
php artisan migrate
```

Horizon provides a beautiful dashboard at `/horizon` with real-time queue metrics.

### Telescope (Optional)

Laravel Telescope provides insights into job execution:

```bash
composer require laravel/telescope --dev
php artisan telescope:install
php artisan migrate
```

Access the dashboard at `/telescope` to view job execution details.

## Related Files

- **Job Classes**: `app/Jobs/RegisterCreatorJob.php`, `app/Jobs/ExecuteSaveTriggerJob.php`
- **Queue Config**: `config/queue.php`
- **Supervisor Config**: `/etc/supervisor/conf.d/laravel-worker.conf`
- **Worker Logs**: `/var/www/html/viksa2/storage/logs/*-worker.log`
- **Controllers**: 
  - `app/Http/Controllers/RegisterCreatorController.php` (dispatches RegisterCreatorJob)
  - `app/Http/Controllers/AdminController.php` (dispatches ExecuteSaveTriggerJob)

## Support

For issues or questions about queue management, refer to:
- Laravel Queue Documentation: https://laravel.com/docs/queues
- Supervisor Documentation: http://supervisord.org/
- Project Documentation: `resources/views/guide/`

---

**Last Updated**: October 10, 2025
**Author**: DevQMS Team