myCRM/docs/GIT_PERFORMANCE_OPTIMIZATION.md

Git Repository API Performance Optimization

Problem

The /api/git-repos/{id}/commits endpoint was taking 66+ seconds (66678ms) to fetch 50 commits from GitHub, making the application unusable.

Root Causes Identified

1. No GitHub Authentication Token

   • Unauthenticated GitHub API requests: 60 requests/hour rate limit
   • Authenticated requests: 5000 requests/hour
   • Without a token, GitHub severely throttles requests causing massive delays

2. No HTTP Timeout Configuration

   • HTTP client waited indefinitely when GitHub throttled requests
   • No retry logic for failed or rate-limited requests
   • No maximum duration limits

3. Cache Working But Not Enough

   • 15-minute cache was configured but doesn't help first-time requests
   • GitHub API calls were still the bottleneck

Solutions Implemented

1. HTTP Client Configuration (config/packages/http_client.yaml)

Default timeout and retry logic:

default_options:
    timeout: 10          # Connection timeout: 10 seconds
    max_duration: 30     # Maximum request duration: 30 seconds
    
    retry_failed:
        max_retries: 2   # Retry failed requests twice
        delay: 1000      # Initial delay: 1 second
        multiplier: 2    # Exponential backoff

Scoped clients for GitHub and Gitea:

scoped_clients:
    github.client:
        base_uri: 'https://api.github.com'
        timeout: 15
        max_duration: 45
        headers:
            Accept: 'application/vnd.github.v3+json'
            User-Agent: 'myCRM-App'
    
    gitea.client:
        scope: 'https?://.*'
        timeout: 15
        max_duration: 45
        headers:
            Accept: 'application/json'
            User-Agent: 'myCRM-App'

2. GitHub Token Configuration

Added to .env:

###> app/git-services ###
# Optional: GitHub Personal Access Token for higher rate limits
# Create at: https://github.com/settings/tokens (needs 'public_repo' scope)
GITHUB_TOKEN=

# Optional: Gitea Access Token for private instances
GITEA_TOKEN=
###< app/git-services ###

How to create a GitHub token:

1. Go to: https://github.com/settings/tokens
2. Click "Generate new token (classic)"
3. Select scopes:
   • public_repo - For accessing public repositories
   • repo - For accessing private repositories (if needed)
4. Copy token and add to .env.local:

   GITHUB_TOKEN=ghp_your_token_here
   

3. Service Configuration Update

Updated config/services.yaml:

App\Service\GitHubService:
    arguments:
        $httpClient: '@github.client'  # Use scoped client
        $githubToken: '%env(string:default::GITHUB_TOKEN)%'

App\Service\GiteaService:
    arguments:
        $httpClient: '@gitea.client'  # Use scoped client
        $giteaToken: '%env(string:default::GITEA_TOKEN)%'

Expected Performance Improvements

Scenario	Before	After	Improvement
First request (no cache)	66+ seconds	2-5 seconds	92-97% faster
Cached requests	66+ seconds	<50ms	99.9% faster
With GitHub token	66+ seconds	1-2 seconds	98% faster
Timeout prevents hanging	∞ (infinite wait)	Max 45 seconds	Guaranteed response

Testing Performance

Test without token:

# First request (cache miss)
curl "https://mycrm.test/api/git-repos/4/commits?branch=main&limit=50" -w "\nTime: %{time_total}s\n"

# Second request (cache hit - should be <50ms)
curl "https://mycrm.test/api/git-repos/4/commits?branch=main&limit=50" -w "\nTime: %{time_total}s\n"

Test with token:

1. Add GITHUB_TOKEN=ghp_xxx to .env.local
2. Clear cache: php bin/console cache:clear
3. Clear API cache via UI or: curl -X POST https://mycrm.test/api/git-repos/4/refresh-cache
4. Test again - should be 1-2 seconds

Monitor rate limits:

curl -I https://api.github.com/rate_limit
# Without token: X-RateLimit-Limit: 60
# With token: X-RateLimit-Limit: 5000

Additional Optimizations

1. Reduce Cache Duration (if needed)

Current: 15 minutes (900 seconds)

// In GitRepositoryController.php line 39
$item->expiresAfter(900); // Reduce to 300 (5 min) for more frequent updates

2. Background Jobs (future enhancement)

For repositories with frequent updates, consider:

• Symfony Messenger to fetch commits in background
• Pre-warm cache on schedule (every 10 minutes)
• Webhook integration with GitHub/Gitea for instant updates

3. Partial Response Loading

Instead of loading all 50 commits at once:

• Load 10 commits initially (fast)
• Lazy-load more on scroll (pagination)

Monitoring

Check cache effectiveness:

# Monitor cache hit/miss ratio
php bin/console cache:pool:list
php bin/console cache:pool:prune  # Clean up old cache entries

Check HTTP client metrics:

# Enable Symfony profiler and check HTTP client panel
# Shows: request duration, retry attempts, cache hits

Rollback Plan

If issues occur, revert by:

1. Remove config/packages/http_client.yaml
2. Restore original config/services.yaml (remove $httpClient arguments)
3. Clear cache: php bin/console cache:clear

Related Files

• config/packages/http_client.yaml - HTTP client configuration
• config/services.yaml - Service injection configuration
• .env - Token configuration (template)
• .env.local - Actual tokens (gitignored)
• src/Controller/GitRepositoryController.php - Caching logic
• src/Service/GitHubService.php - GitHub API integration
• src/Service/GiteaService.php - Gitea API integration