citrus-cms/resources/views/guide/batch-excel-save-documentation.md

Batch Excel Save Documentation

Overview

The batch Excel save functionality allows users to import large amounts of data from Excel files into the database. This system processes Excel data row by row, handles different data types, manages permissions, and provides detailed feedback about the import process.

How It Works

1. Initial Setup

The system starts by setting up the environment for handling large files:

• Time Limits: Removes time restrictions to handle large files
• File Size: Increases upload limits to 2GB
• Memory: Allocates 8GB of memory for processing
• Execution Time: Allows unlimited processing time

2. Data Type Detection

Before processing, the system automatically detects what type of data each column contains:

• Numeric Columns: Identifies columns that should contain numbers (integers, decimals, etc.)
• Date Columns: Recognizes date fields and special date columns
• Text Columns: Handles regular text data

3. Permission Checking

The system checks user permissions before processing:

• Delete Permission: Can the user delete records?
• Modify Permission: Can the user update existing records?
• Write Permission: Can the user create new records?
• Full Control: Can the user do all operations?

4. Data Processing

For each row in the Excel file, the system:

Step 1: Data Validation

• Checks if the row data is properly formatted
• Validates that all required columns are present
• Ensures data structure is correct

Step 2: Data Cleaning

• Removes extra spaces from text
• Converts empty cells to null values
• Handles special values like "NULL*" (converts to null)

Step 3: Data Type Conversion

For Numbers:

• Converts Turkish number format (1.234,56) to standard format (1234.56)
• Handles negative numbers properly
• Removes thousand separators
• Validates numeric values

For Dates:

• Converts various date formats to standard Y-m-d format
• Handles different separators (/, .)
• Uses Carbon library for date parsing
• Validates date values

For Text:

• Trims whitespace
• Handles empty values

5. Record Processing Logic

The system uses different strategies based on the table type:

Strategy 1: ISO Number + Joint Number (for weld logs)

• Looks for existing records using ISO number and joint number
• Updates existing records or creates new ones
• Prevents duplicate entries

Strategy 2: Spool Number + ISO Number

• Uses spool number and ISO number to find existing records
• Updates or creates records accordingly

Strategy 3: Line Number + Fluid Code (for line lists)

• Matches records using line number and fluid code
• Updates existing or creates new records

Strategy 4: Test Package Number (for test packages)

• Uses test package number as the unique identifier
• Updates existing or creates new records

Strategy 5: ID-based Processing (default)

• Uses the ID field to determine if record exists
• Updates existing records or creates new ones

6. Duplicate Prevention

For weld logs specifically, the system:

• Checks for duplicate ISO number + joint number combinations
• Prevents creation of duplicate entries
• Reports duplicate errors to the user

7. Memory Management

To handle large files efficiently:

• Processes data in batches of 1000 rows
• Commits database transactions every 1000 rows
• Cleans up memory every 500 rows
• Uses garbage collection to free memory

8. Error Handling

The system tracks different types of issues:

• Processing Errors: Problems with data format or validation
• Permission Errors: User doesn't have required permissions
• Duplicate Errors: Attempted to create duplicate records
• Database Errors: Problems with database operations

9. Change Tracking

For each record processed, the system:

• Records which columns were changed
• Creates audit trail entries
• Tracks who made the changes
• Records the IP address of the user

10. Trigger Execution

After processing all data, the system can run triggers:

• Updates related tables
• Runs business logic
• Executes custom scripts for specific tables

Response Format

The system returns detailed information about the import process:

{
    "storedData": 1500,           // Total records processed
    "update": 800,                // Records updated
    "insert": 700,                // Records created
    "delete": 0,                  // Records deleted
    "error": null,                // Any system errors
    "insertedRowNumbers": [1,2,3], // Excel row numbers for new records
    "updatedRowNumbers": [4,5,6],  // Excel row numbers for updated records
    "errorRows": [],              // Rows with errors
    "totalErrors": 0              // Total number of errors
}

Special Features

1. Row Number Tracking

• Tracks which Excel row numbers were processed
• Reports which rows were inserted, updated, or had errors
• Helps users identify problems in their Excel files

2. Batch Processing

• Processes large files without memory issues
• Commits data in chunks to prevent data loss
• Provides progress feedback

3. Data Validation

• Validates data types before processing
• Converts formats automatically
• Reports invalid data clearly

4. Audit Trail

• Records all changes made
• Tracks who made changes
• Maintains history of modifications

Usage Examples

Basic Import

1. User uploads Excel file
2. System detects data types automatically
3. Processes each row according to table rules
4. Returns detailed results

Error Handling

1. System encounters invalid data
2. Records error with row number
3. Continues processing other rows
4. Reports all errors at the end

Large File Processing

1. System processes 1000 rows at a time
2. Commits data to database
3. Cleans up memory
4. Continues with next batch

Best Practices

1. Prepare Your Data: Ensure Excel columns match database fields
2. Check Permissions: Make sure you have required access rights
3. Validate Data: Check for obvious errors before importing
4. Use Correct Format: Follow the expected data formats
5. Monitor Results: Review the response for any errors

Common Issues and Solutions

Issue: "Invalid data structure"

Solution: Ensure Excel file has correct column headers and data format

Issue: "Permission denied"

Solution: Contact administrator to get required permissions

Issue: "Duplicate entry"

Solution: Check for duplicate records in your Excel file

Issue: "Memory limit exceeded"

Solution: Process smaller batches or contact administrator to increase limits