# Authentication Source: https://docs.cerevox.ai/account/authentication Login, token management, and secure session handling # Authentication Manage user authentication, access tokens, and secure sessions with the Account API. ## Authentication Methods Cerevox supports two authentication methods: **Best for**: Server-side applications, scripts, automation ```python theme={null} from cerevox import Hippo, Lexa # Use API key directly hippo = Hippo(api_key="your-api-key") lexa = Lexa(api_key="your-api-key") ``` **Get API key**: [cerevox.ai](https://cerevox.ai) **Best for**: User-facing applications, mobile apps, web apps ```python theme={null} from cerevox import Account account = Account() # Login to get tokens auth = account.login(email, password) # Use access token hippo = Hippo(access_token=auth.access_token) ``` **Use case**: Multi-user applications ## OAuth Authentication Flow ### 1. Login ```python Sync theme={null} from cerevox import Account account = Account() # Login with email/password auth_response = account.login( email="user@example.com", password="secure-password" ) print(f"Access Token: {auth_response.access_token}") print(f"Refresh Token: {auth_response.refresh_token}") print(f"Expires In: {auth_response.expires_in} seconds") print(f"Token Type: {auth_response.token_type}") ``` ```python Async theme={null} from cerevox import AsyncAccount async with AsyncAccount() as account: auth = await account.login( email="user@example.com", password="password" ) print(f"Access Token: {auth.access_token}") ``` **Response fields:** * `access_token`: Short-lived token for API requests (1 hour) * `refresh_token`: Long-lived token to get new access tokens (30 days) * `expires_in`: Seconds until access token expires * `token_type`: Token type (usually "Bearer") ### 2. Use Access Token ```python theme={null} from cerevox import Hippo, Lexa # Use access token with any Cerevox API hippo = Hippo(access_token=auth_response.access_token) lexa = Lexa(access_token=auth_response.access_token) # Make API calls folder = hippo.create_folder("My Docs") documents = lexa.parse(["doc.pdf"]) ``` ### 3. Refresh Token Access tokens expire after 1 hour. Refresh before expiry: ```python Sync theme={null} # Refresh to get new access token new_auth = account.refresh_token( refresh_token=auth_response.refresh_token ) print(f"New Access Token: {new_auth.access_token}") print(f"Expires In: {new_auth.expires_in} seconds") ``` ```python Async theme={null} new_auth = await account.refresh_token( refresh_token=auth_response.refresh_token ) ``` ### 4. Revoke Token (Logout) ```python Sync theme={null} # Revoke token to log out account.revoke_token(token=auth_response.access_token) print("Logged out successfully") ``` ```python Async theme={null} await account.revoke_token(token=auth_response.access_token) ``` After revoking, the access token can no longer be used for API requests. ## Complete Authentication Flow ```python theme={null} from cerevox import Account, Hippo import time class AuthManager: def __init__(self): self.account = Account() self.auth_response = None def login(self, email, password): """Login and store tokens""" self.auth_response = self.account.login(email, password) # Calculate expiry time self.auth_response.expires_at = ( time.time() + self.auth_response.expires_in ) print(f"✅ Logged in as {email}") return self.auth_response def get_valid_token(self): """Get valid access token, refreshing if needed""" # Check if token expires within 5 minutes if time.time() > self.auth_response.expires_at - 300: print("🔄 Refreshing token...") self.auth_response = self.account.refresh_token( self.auth_response.refresh_token ) self.auth_response.expires_at = ( time.time() + self.auth_response.expires_in ) print("✅ Token refreshed") return self.auth_response.access_token def logout(self): """Revoke token and log out""" self.account.revoke_token(self.auth_response.access_token) self.auth_response = None print("✅ Logged out") # Usage auth_manager = AuthManager() # Login auth_manager.login("user@example.com", "password") # Use with Hippo hippo = Hippo(access_token=auth_manager.get_valid_token()) folder = hippo.create_folder("My Docs") # Token automatically refreshes when needed time.sleep(3600) # Wait 1 hour hippo = Hippo(access_token=auth_manager.get_valid_token()) # Auto-refreshes # Logout when done auth_manager.logout() ``` ## Secure Token Storage **Never** store tokens in: * Source code * Version control (git) * Client-side storage (localStorage, cookies without security) * Plain text files ### Recommended: Environment Variables ```python theme={null} import os # Store tokens in environment variables os.environ['CEREVOX_ACCESS_TOKEN'] = auth_response.access_token os.environ['CEREVOX_REFRESH_TOKEN'] = auth_response.refresh_token # Retrieve when needed access_token = os.getenv('CEREVOX_ACCESS_TOKEN') ``` ### Recommended: Secure Keyring ```python theme={null} import keyring # Store tokens securely in system keyring keyring.set_password("cerevox", "access_token", auth_response.access_token) keyring.set_password("cerevox", "refresh_token", auth_response.refresh_token) # Retrieve when needed access_token = keyring.get_password("cerevox", "access_token") refresh_token = keyring.get_password("cerevox", "refresh_token") ``` ### Server-Side Sessions ```python theme={null} from flask import Flask, session app = Flask(__name__) app.secret_key = 'your-secret-key' # Use secure random key @app.route('/login', methods=['POST']) def login(): # Login auth = account.login(email, password) # Store in server-side session session['access_token'] = auth.access_token session['refresh_token'] = auth.refresh_token session['expires_at'] = time.time() + auth.expires_in return "Logged in" @app.route('/api/data') def get_data(): # Use token from session access_token = session.get('access_token') hippo = Hippo(access_token=access_token) # ... use hippo ... ``` ## Error Handling ```python theme={null} from cerevox import Account, AccountError account = Account() try: auth = account.login(email="user@example.com", password="wrong-password") except AccountError as e: if "invalid credentials" in str(e).lower(): print("Error: Invalid email or password") elif "account locked" in str(e).lower(): print("Error: Account locked due to too many failed attempts") elif "not found" in str(e).lower(): print("Error: Account not found") else: print(f"Error: {e}") # Token expiry handling try: hippo = Hippo(access_token=expired_token) folder = hippo.create_folder("Test") except AccountError as e: if "token expired" in str(e).lower(): # Refresh token new_auth = account.refresh_token(refresh_token) hippo = Hippo(access_token=new_auth.access_token) # Retry operation folder = hippo.create_folder("Test") ``` ## Token Lifecycle User provides credentials → Receive access + refresh tokens Include access token in API requests (valid for 1 hour) Before expiry, use refresh token to get new access token Revoke access token when user logs out ```mermaid theme={null} stateDiagram-v2 [*] --> Logged Out Logged Out --> Logged In: login() Logged In --> API Requests: Use access_token API Requests --> Token Refresh: Token expires Token Refresh --> API Requests: refresh_token() API Requests --> Logged Out: revoke_token() Logged Out --> [*] ``` ## Multi-User Application Example ```python theme={null} from cerevox import Account, Hippo from flask import Flask, request, session, jsonify app = Flask(__name__) app.secret_key = 'secure-random-key' account = Account() @app.route('/auth/login', methods=['POST']) def login(): """User login endpoint""" data = request.json try: auth = account.login( email=data['email'], password=data['password'] ) # Store in session session['access_token'] = auth.access_token session['refresh_token'] = auth.refresh_token session['expires_at'] = time.time() + auth.expires_in return jsonify({'message': 'Logged in successfully'}) except Exception as e: return jsonify({'error': str(e)}), 401 @app.route('/auth/logout', methods=['POST']) def logout(): """User logout endpoint""" try: # Revoke token account.revoke_token(session['access_token']) # Clear session session.clear() return jsonify({'message': 'Logged out successfully'}) except Exception as e: return jsonify({'error': str(e)}), 400 def get_valid_token(): """Helper to get valid access token""" # Check if token expires soon if time.time() > session.get('expires_at', 0) - 300: # Refresh token auth = account.refresh_token(session['refresh_token']) # Update session session['access_token'] = auth.access_token session['expires_at'] = time.time() + auth.expires_in return session['access_token'] @app.route('/api/folders', methods=['POST']) def create_folder(): """Protected endpoint - requires authentication""" try: # Get valid token access_token = get_valid_token() # Use with Hippo hippo = Hippo(access_token=access_token) folder = hippo.create_folder(request.json['name']) return jsonify({ 'id': folder.id, 'name': folder.name }) except Exception as e: return jsonify({'error': str(e)}), 400 if __name__ == '__main__': app.run() ``` ## Best Practices Refresh tokens **before** they expire: ```python theme={null} # ✅ Good - Refresh 5 minutes before expiry if time.time() > expires_at - 300: auth = account.refresh_token(refresh_token) # ❌ Bad - Wait until token expires try: hippo.create_folder("Test") except: auth = account.refresh_token(refresh_token) ``` Prevents failed requests due to expiry ```python theme={null} # ✅ Good password = os.getenv('USER_PASSWORD') access_token = keyring.get_password("cerevox", "token") # ❌ Bad password = "hardcoded-password" access_token = "sk_live_abc123..." ``` Never hardcode credentials! Always provide logout functionality: ```python theme={null} def logout(): # Revoke token account.revoke_token(access_token) # Clear stored tokens session.clear() # or keyring.delete_password("cerevox", "access_token") ``` Prevents unauthorized access after user leaves ```python theme={null} try: auth = account.refresh_token(refresh_token) except AccountError: # Refresh token expired or revoked # Force user to log in again redirect_to_login() ``` Gracefully handle expired refresh tokens ## Security Checklist * [ ] Use HTTPS for all API requests * [ ] Never commit API keys or tokens to version control * [ ] Store tokens securely (keyring, encrypted storage) * [ ] Implement token refresh before expiry * [ ] Provide logout functionality * [ ] Handle authentication errors gracefully * [ ] Use environment variables for secrets * [ ] Implement rate limiting on login endpoint * [ ] Log authentication events for security monitoring ## Next Steps Monitor API usage and billing Back to Account API overview # Account Management Source: https://docs.cerevox.ai/account/overview Enterprise authentication, user management, and usage tracking # Account Management API Manage authentication, users, and usage tracking for your Cerevox account. ## What is the Account API? The **Account API** provides enterprise-grade account operations: Login, refresh tokens, and secure session management Create, update, and manage users (admin operations) Monitor API usage, costs, and billing information ## Core Features ### Authentication & Tokens ```python Login theme={null} from cerevox import Account account = Account() # Login with credentials response = account.login( email="user@example.com", password="secure-password" ) print(f"Access Token: {response.access_token}") print(f"Refresh Token: {response.refresh_token}") print(f"Expires in: {response.expires_in} seconds") ``` ```python Refresh Token theme={null} # Refresh access token before expiry new_tokens = account.refresh_token( refresh_token=response.refresh_token ) print(f"New Access Token: {new_tokens.access_token}") ``` ```python Revoke Token theme={null} # Revoke token (logout) account.revoke_token(token=response.access_token) print("Token revoked successfully") ``` ### Account Information ```python Get Account Info theme={null} # Get account details info = account.get_account_info() print(f"Account ID: {info.account_id}") print(f"Email: {info.email}") print(f"Plan: {info.plan}") print(f"Status: {info.status}") ``` ```python Async theme={null} from cerevox import AsyncAccount async with AsyncAccount() as account: info = await account.get_account_info() print(f"Plan: {info.plan}") ``` ### Usage Monitoring ```python Check Usage theme={null} # Get usage statistics usage = account.get_usage() print(f"API Calls: {usage.total_requests}") print(f"Documents Processed: {usage.documents_processed}") print(f"Questions Asked: {usage.questions_asked}") print(f"Storage Used: {usage.storage_bytes} bytes") ``` ```python Get Billing Info theme={null} # Check billing and costs billing = account.get_billing_info() print(f"Current Period: {billing.period_start} to {billing.period_end}") print(f"Total Cost: ${billing.total_cost}") print(f"Next Invoice: {billing.next_invoice_date}") ``` ## User Management (Admin) **Admin only**: User management operations require admin privileges. ```python Create User theme={null} # Create new user (admin only) new_user = account.create_user( email="newuser@example.com", role="member", name="New User" ) print(f"Created user: {new_user.email}") ``` ```python List Users theme={null} # Get all users in account (admin only) users = account.list_users() for user in users: print(f"{user.name} ({user.email}) - {user.role}") ``` ```python Update User theme={null} # Update user role (admin only) updated_user = account.update_user( user_id=user.id, role="admin" ) print(f"Updated {updated_user.email} to {updated_user.role}") ``` ```python Delete User theme={null} # Delete user (admin only) account.delete_user(user_id=user.id) print("User deleted successfully") ``` ## API Clients ```python Synchronous theme={null} from cerevox import Account # Best for: Scripts, notebooks, simple applications account = Account(api_key="your-api-key") info = account.get_account_info() usage = account.get_usage() ``` ```python Asynchronous theme={null} from cerevox import AsyncAccount import asyncio # Best for: Web servers, high-performance apps async def main(): async with AsyncAccount(api_key="your-api-key") as account: info = await account.get_account_info() usage = await account.get_usage() print(f"Plan: {info.plan}") print(f"API Calls: {usage.total_requests}") asyncio.run(main()) ``` ## Common Use Cases ```python theme={null} from cerevox import Account account = Account() # Check usage usage = account.get_usage() # Alert if approaching limits if usage.total_requests > usage.rate_limit * 0.8: print("⚠️ Warning: Approaching rate limit!") print(f"Used: {usage.total_requests} / {usage.rate_limit}") # Check costs billing = account.get_billing_info() print(f"Current period cost: ${billing.total_cost}") ``` **Benefit**: Proactive monitoring prevents surprises ```python theme={null} # Add team members (admin only) team_members = [ {"email": "dev1@company.com", "role": "member"}, {"email": "dev2@company.com", "role": "member"}, {"email": "manager@company.com", "role": "admin"} ] for member in team_members: user = account.create_user( email=member["email"], role=member["role"] ) print(f"Added: {user.email}") ``` **Benefit**: Centralized team access management ```python theme={null} # Login and store tokens securely auth = account.login(email, password) # Store in secure storage (not in code!) import keyring keyring.set_password("cerevox", "access_token", auth.access_token) keyring.set_password("cerevox", "refresh_token", auth.refresh_token) # Retrieve when needed access_token = keyring.get_password("cerevox", "access_token") ``` **Benefit**: Secure credential management ```python theme={null} import pandas as pd # Get usage over time usage_data = [] for month in range(1, 13): usage = account.get_usage( start_date=f"2025-{month:02d}-01", end_date=f"2025-{month:02d}-28" ) usage_data.append({ 'month': month, 'requests': usage.total_requests, 'cost': usage.estimated_cost }) df = pd.DataFrame(usage_data) print(df) ``` **Benefit**: Track trends and optimize usage ## Authentication Flow ```mermaid theme={null} sequenceDiagram participant App participant Account API participant Cerevox Services App->>Account API: login(email, password) Account API->>App: access_token, refresh_token App->>Cerevox Services: API call (with access_token) Cerevox Services->>App: Response Note over App: Token expires App->>Account API: refresh_token() Account API->>App: new access_token App->>Cerevox Services: API call (with new token) Cerevox Services->>App: Response App->>Account API: revoke_token() [logout] Account API->>App: Success ``` ## Response Models ### Account Info ```python theme={null} { 'account_id': 'acc_123', 'email': 'user@example.com', 'name': 'John Doe', 'plan': 'pro', # free, pro, enterprise 'status': 'active', # active, suspended, cancelled 'created_at': '2025-01-01T00:00:00Z', 'features': { 'hippo_enabled': True, 'lexa_enabled': True, 'max_users': 10 } } ``` ### Usage Stats ```python theme={null} { 'total_requests': 15420, 'documents_processed': 230, 'questions_asked': 1250, 'storage_bytes': 524288000, # ~500MB 'rate_limit': 100000, 'rate_limit_reset': '2025-02-01T00:00:00Z' } ``` ### Billing Info ```python theme={null} { 'period_start': '2025-01-01', 'period_end': '2025-01-31', 'total_cost': 125.50, 'currency': 'USD', 'breakdown': { 'hippo_cost': 75.00, 'lexa_cost': 45.00, 'storage_cost': 5.50 }, 'next_invoice_date': '2025-02-01' } ``` ## Best Practices **Never** hardcode credentials: ```python theme={null} # ❌ Bad account = Account(api_key="sk_live_abc123...") # ✅ Good - Use environment variables import os account = Account(api_key=os.getenv("CEREVOX_API_KEY")) ``` **Never** commit credentials to version control! ```python theme={null} # Proactively refresh before expiry def get_valid_token(auth_response): import time # Check if token expires soon (within 5 minutes) if time.time() > auth_response.expires_at - 300: # Refresh token auth_response = account.refresh_token( auth_response.refresh_token ) return auth_response.access_token ``` ```python theme={null} # Daily usage check def check_daily_usage(): usage = account.get_usage() # Check limits usage_percent = usage.total_requests / usage.rate_limit * 100 if usage_percent > 80: notify_team(f"Usage at {usage_percent:.0f}%") # Schedule daily import schedule schedule.every().day.at("09:00").do(check_daily_usage) ``` ## Error Handling ```python theme={null} from cerevox import Account, AccountError account = Account() try: info = account.get_account_info() print(f"Plan: {info.plan}") except AccountError as e: if "authentication" in str(e).lower(): print("Error: Invalid credentials") elif "forbidden" in str(e).lower(): print("Error: Insufficient permissions") else: print(f"Error: {e}") ``` ## Next Steps Complete authentication guide Monitor usage and billing Use Hippo for RAG & retrieval Use Lexa for document parsing # Usage Tracking Source: https://docs.cerevox.ai/account/usage-tracking Monitor API usage, costs, and billing information # Usage Tracking Monitor your API usage, track costs, and manage billing with the Account API. ## Overview Track three key metrics: Total API calls, documents processed, questions asked Storage used, rate limits, quotas Current period costs, next invoice, breakdown ## Get Usage Statistics ```python Sync theme={null} from cerevox import Account account = Account(api_key="your-api-key") # Get current usage usage = account.get_usage() print(f"API Calls: {usage.total_requests}") print(f"Documents Processed: {usage.documents_processed}") print(f"Questions Asked: {usage.questions_asked}") print(f"Storage Used: {usage.storage_bytes / (1024**3):.2f} GB") print(f"Rate Limit: {usage.rate_limit}") ``` ```python Async theme={null} from cerevox import AsyncAccount async with AsyncAccount(api_key="your-api-key") as account: usage = await account.get_usage() print(f"Total Requests: {usage.total_requests}") print(f"Remaining: {usage.rate_limit - usage.total_requests}") ``` **Response fields:** * `total_requests`: Total API calls in current period * `documents_processed`: Total documents processed * `questions_asked`: Total questions submitted to Hippo * `storage_bytes`: Total storage used in bytes * `rate_limit`: Maximum requests allowed per period * `rate_limit_reset`: When rate limit resets (timestamp) ## Usage by Date Range ```python theme={null} from datetime import datetime, timedelta # Get usage for specific date range start_date = datetime(2025, 1, 1) end_date = datetime(2025, 1, 31) usage = account.get_usage( start_date=start_date.isoformat(), end_date=end_date.isoformat() ) print(f"January 2025 Usage:") print(f" API Calls: {usage.total_requests}") print(f" Documents: {usage.documents_processed}") print(f" Questions: {usage.questions_asked}") ``` ## Billing Information ```python Get Billing Info theme={null} # Get current billing period information billing = account.get_billing_info() print(f"Billing Period: {billing.period_start} to {billing.period_end}") print(f"Total Cost: ${billing.total_cost:.2f}") print(f"Currency: {billing.currency}") # Cost breakdown print("\nCost Breakdown:") print(f" Hippo: ${billing.breakdown.hippo_cost:.2f}") print(f" Lexa: ${billing.breakdown.lexa_cost:.2f}") print(f" Storage: ${billing.breakdown.storage_cost:.2f}") print(f"\nNext Invoice: {billing.next_invoice_date}") ``` ```python Async theme={null} billing = await account.get_billing_info() print(f"Total: ${billing.total_cost}") print(f"Next Invoice: {billing.next_invoice_date}") ``` **Response fields:** * `period_start`: Billing period start date * `period_end`: Billing period end date * `total_cost`: Total cost for period * `currency`: Currency code (USD, EUR, etc.) * `breakdown`: Cost breakdown by service * `next_invoice_date`: When next invoice is generated ## Monitor Rate Limits ```python theme={null} # Check rate limit status usage = account.get_usage() # Calculate usage percentage usage_percent = (usage.total_requests / usage.rate_limit) * 100 print(f"Rate Limit Status:") print(f" Used: {usage.total_requests:,} / {usage.rate_limit:,}") print(f" Percentage: {usage_percent:.1f}%") print(f" Resets: {usage.rate_limit_reset}") # Alert if approaching limit if usage_percent > 80: print("\n⚠️ WARNING: Approaching rate limit!") print("Consider upgrading plan or optimizing usage") elif usage_percent > 90: print("\n🚨 CRITICAL: Nearly at rate limit!") print("Immediate action required") ``` ## Usage Monitoring Dashboard ```python theme={null} from cerevox import Account from datetime import datetime, timedelta import pandas as pd class UsageMonitor: def __init__(self, api_key): self.account = Account(api_key=api_key) def get_daily_summary(self): """Get usage summary for today""" usage = self.account.get_usage() return { 'api_calls': usage.total_requests, 'documents': usage.documents_processed, 'questions': usage.questions_asked, 'storage_gb': usage.storage_bytes / (1024**3), 'rate_limit_used_pct': (usage.total_requests / usage.rate_limit) * 100 } def get_weekly_trend(self): """Get usage trend for past 7 days""" end_date = datetime.now() start_date = end_date - timedelta(days=7) usage_data = [] for i in range(7): day = start_date + timedelta(days=i) usage = self.account.get_usage( start_date=day.isoformat(), end_date=day.isoformat() ) usage_data.append({ 'date': day.strftime('%Y-%m-%d'), 'api_calls': usage.total_requests, 'cost': usage.estimated_cost }) return pd.DataFrame(usage_data) def check_alerts(self): """Check for usage alerts""" usage = self.account.get_usage() billing = self.account.get_billing_info() alerts = [] # Rate limit check rate_limit_pct = (usage.total_requests / usage.rate_limit) * 100 if rate_limit_pct > 80: alerts.append({ 'type': 'rate_limit', 'severity': 'warning' if rate_limit_pct < 90 else 'critical', 'message': f'Rate limit at {rate_limit_pct:.0f}%' }) # Cost check (example threshold: $100) if billing.total_cost > 100: alerts.append({ 'type': 'cost', 'severity': 'warning', 'message': f'Monthly cost: ${billing.total_cost:.2f}' }) return alerts # Usage monitor = UsageMonitor(api_key="your-api-key") # Daily summary summary = monitor.get_daily_summary() print("Today's Usage:") print(f" API Calls: {summary['api_calls']:,}") print(f" Documents: {summary['documents']:,}") print(f" Questions: {summary['questions']:,}") # Weekly trend trend = monitor.get_weekly_trend() print("\nWeekly Trend:") print(trend) # Alerts alerts = monitor.check_alerts() if alerts: print("\n🚨 Alerts:") for alert in alerts: print(f" [{alert['severity'].upper()}] {alert['message']}") ``` ## Cost Analysis ### Understand Cost Breakdown ```python theme={null} billing = account.get_billing_info() # Total cost total = billing.total_cost # Service breakdown hippo_pct = (billing.breakdown.hippo_cost / total) * 100 lexa_pct = (billing.breakdown.lexa_cost / total) * 100 storage_pct = (billing.breakdown.storage_cost / total) * 100 print("Cost Breakdown:") print(f" Hippo (RAG): ${billing.breakdown.hippo_cost:8.2f} ({hippo_pct:.1f}%)") print(f" Lexa (Parse): ${billing.breakdown.lexa_cost:8.2f} ({lexa_pct:.1f}%)") print(f" Storage: ${billing.breakdown.storage_cost:8.2f} ({storage_pct:.1f}%)") print(f" {'─' * 40}") print(f" Total: ${total:8.2f}") ``` ### Calculate ROI from 80% Savings ```python theme={null} # Traditional RAG cost (without Hippo) questions_asked = usage.questions_asked traditional_cost_per_question = 0.05 # $0.05 per query traditional_total = questions_asked * traditional_cost_per_question # Actual cost with Hippo actual_hippo_cost = billing.breakdown.hippo_cost # Savings savings = traditional_total - actual_hippo_cost savings_percent = (savings / traditional_total) * 100 print("Cost Comparison:") print(f" Traditional RAG: ${traditional_total:8.2f}") print(f" With Hippo: ${actual_hippo_cost:8.2f}") print(f" {'─' * 40}") print(f" Savings: ${savings:8.2f} ({savings_percent:.0f}%)") ``` ## Usage Optimization ```python theme={null} # ❌ Bad - Create new chat for every question for question in questions: chat = hippo.create_chat(folder.id, "Q&A") answer = hippo.submit_ask(chat.id, question) hippo.delete_chat(chat.id) # ✅ Good - Reuse chat for related questions chat = hippo.create_chat(folder.id, "Q&A Session") for question in questions: answer = hippo.submit_ask(chat.id, question) ``` **Impact**: Reduces overhead, maintains context ```python theme={null} import asyncio from cerevox import AsyncHippo # ✅ Good - Concurrent upload async with AsyncHippo() as hippo: tasks = [hippo.upload_file(folder.id, f) for f in files] await asyncio.gather(*tasks) ``` **Impact**: Faster, more efficient processing ```python theme={null} # Delete old/unused folders folders = hippo.get_folders() for folder in folders: if folder.file_count == 0: hippo.delete_folder(folder.id) ``` **Impact**: Reduces storage costs ```python theme={null} # ❌ Bad - Vague question answer = hippo.submit_ask(chat.id, "Tell me about features") # ✅ Good - Specific question answer = hippo.submit_ask( chat.id, "What are the three main features of Product X?" ) ``` **Impact**: Better answers with less retrieval overhead ## Set Up Usage Alerts ```python theme={null} import schedule import time from cerevox import Account account = Account() def check_usage_alerts(): """Check usage and send alerts""" usage = account.get_usage() # Rate limit alert usage_pct = (usage.total_requests / usage.rate_limit) * 100 if usage_pct > 80: send_alert( title="Rate Limit Warning", message=f"Usage at {usage_pct:.0f}% of rate limit", severity="warning" ) # Cost alert billing = account.get_billing_info() if billing.total_cost > 100: # Threshold: $100 send_alert( title="Cost Alert", message=f"Monthly cost: ${billing.total_cost:.2f}", severity="info" ) def send_alert(title, message, severity): """Send alert via email, Slack, etc.""" print(f"[{severity.upper()}] {title}: {message}") # Implement your notification logic here # - Send email # - Post to Slack # - Log to monitoring system # Schedule daily checks schedule.every().day.at("09:00").do(check_usage_alerts) # Run scheduler while True: schedule.run_pending() time.sleep(60) ``` ## Export Usage Data ```python theme={null} import csv from datetime import datetime, timedelta def export_usage_report(account, days=30): """Export usage data to CSV""" end_date = datetime.now() start_date = end_date - timedelta(days=days) # Collect daily usage usage_data = [] current_date = start_date while current_date <= end_date: usage = account.get_usage( start_date=current_date.isoformat(), end_date=current_date.isoformat() ) usage_data.append({ 'date': current_date.strftime('%Y-%m-%d'), 'api_calls': usage.total_requests, 'documents': usage.documents_processed, 'questions': usage.questions_asked, 'cost': usage.estimated_cost }) current_date += timedelta(days=1) # Write to CSV with open('usage_report.csv', 'w', newline='') as f: writer = csv.DictWriter( f, fieldnames=['date', 'api_calls', 'documents', 'questions', 'cost'] ) writer.writeheader() writer.writerows(usage_data) print(f"✅ Exported {len(usage_data)} days to usage_report.csv") # Usage export_usage_report(account, days=30) ``` ## Plan Limits **Rate Limits:** * 1,000 API calls/month * 100 documents/month * 500 questions/month **Storage:** * 1 GB included **Features:** * Hippo, Lexa, Account APIs * 1 user **Rate Limits:** * 100,000 API calls/month * 10,000 documents/month * 50,000 questions/month **Storage:** * 100 GB included **Features:** * All Free features * Up to 10 users * Priority support **Rate Limits:** * Custom limits **Storage:** * Unlimited **Features:** * All Pro features * Unlimited users * SLA guarantee * Dedicated support * Custom integrations ## Best Practices Monitor usage daily to catch issues early: * Track rate limit usage * Monitor costs * Review usage trends * Set up automated alerts Look for patterns in your usage: * Peak usage times * Cost trends over time * Most expensive operations * Optimization opportunities Reduce costs without sacrificing quality: * Use async for batch operations * Reuse chats and folders * Clean up unused resources * Write specific questions Forecast usage and costs: * Estimate monthly costs * Plan for growth * Upgrade plan proactively * Budget for peak periods ## Next Steps Manage authentication and tokens Back to Account API overview Optimize RAG costs and quality # Lexa Client Source: https://docs.cerevox.ai/api/client Complete reference for initializing and configuring the Lexa client ## Client Initialization ### Synchronous Client ```python Basic Setup theme={null} from cerevox import Lexa # Initialize with API key client = Lexa(api_key="your-api-key") # Parse documents documents = client.parse(["document.pdf"]) ``` ```python Environment Variable theme={null} import os from cerevox import Lexa # Set environment variable: CEREVOX_API_KEY client = Lexa() # Automatically uses CEREVOX_API_KEY documents = client.parse(["document.pdf"]) ``` ```python Custom Configuration theme={null} from cerevox import Lexa client = Lexa( api_key="your-api-key", base_url="https://data.cerevox.ai", # Custom endpoint timeout=120.0, # Request timeout max_retries=3, # Retry attempts retry_delay=1.0 # Delay between retries ) ``` ### Asynchronous Client ```python Context Manager (Recommended) theme={null} import asyncio from cerevox import AsyncLexa async def main(): async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(["document.pdf"]) return documents asyncio.run(main()) ``` ```python Manual Management theme={null} import asyncio from cerevox import AsyncLexa async def main(): client = AsyncLexa(api_key="your-api-key") try: documents = await client.parse(["document.pdf"]) return documents finally: await client.close() asyncio.run(main()) ``` ## Client Configuration ### Parameters Your Cerevox API key. Get one at [cerevox.ai/lexa](https://cerevox.ai/lexa) Base URL for the Cerevox API endpoint Request timeout in seconds for API calls Maximum number of retry attempts for failed requests Delay in seconds between retry attempts ### Environment Variables ```bash Environment Setup theme={null} # Required export CEREVOX_API_KEY="your-api-key" # Optional overrides export CEREVOX_BASE_URL="https://data.cerevox.ai" export CEREVOX_TIMEOUT="120" export CEREVOX_MAX_RETRIES="5" ``` ```python Python Usage theme={null} from cerevox import Lexa # Automatically uses environment variables client = Lexa() ``` ## Client Methods ### Core Parsing Methods Parse local files or file-like objects. ```python theme={null} documents = client.parse( files=["document.pdf", "report.docx"], mode=ProcessingMode.DEFAULT, progress_callback=None, timeout=60.0, poll_interval=2.0 ) ``` Parse files from URLs. ```python theme={null} documents = client.parse_urls( urls=["https://example.com/document.pdf"], mode=ProcessingMode.DEFAULT, progress_callback=None, timeout=120.0, poll_interval=2.0 ) ``` Get the current status of a parsing job. ```python theme={null} status = client.get_job_status(job_id="job_123") print(f"Status: {status.status}") ``` ### Cloud Storage Methods ```python theme={null} # List S3 buckets buckets = client.list_s3_buckets() # List S3 folder contents contents = client.list_s3_folder("bucket-name", "folder-path/") # Parse S3 folder documents = client.parse_s3_folder( bucket="bucket-name", folder_path="documents/", mode=ProcessingMode.DEFAULT ) ``` ```python theme={null} # List SharePoint sites sites = client.list_sharepoint_sites() # List drives in a site drives = client.list_sharepoint_drives("site-id") # Parse SharePoint folder documents = client.parse_sharepoint_folder( site_id="site-id", drive_id="drive-id", folder_path="Documents/", mode=ProcessingMode.DEFAULT ) ``` ```python theme={null} # List Box folders folders = client.list_box_folders(parent_folder_id="0") # Parse Box folder documents = client.parse_box_folder( folder_id="123456789", mode=ProcessingMode.DEFAULT ) ``` ## Processing Modes Choose the right processing mode for your use case: **Fast and efficient** * Optimized for speed * Good accuracy * Lower resource usage * Recommended for most use cases **Maximum accuracy** * Highest accuracy * Enhanced table extraction * More thorough analysis * Best for complex documents ```python Processing Mode Usage theme={null} from cerevox import Lexa, ProcessingMode client = Lexa(api_key="your-api-key") # Default mode (recommended) - fast and efficient documents = client.parse(["document.pdf"], mode=ProcessingMode.DEFAULT) # Advanced mode for maximum accuracy documents = client.parse(["document.pdf"], mode=ProcessingMode.ADVANCED) ``` ## Error Handling The Lexa client provides comprehensive error handling: ```python Basic Error Handling theme={null} from cerevox import Lexa, LexaError client = Lexa(api_key="your-api-key") try: documents = client.parse(["document.pdf"]) print(f"Successfully parsed {len(documents)} documents") except LexaError as e: print(f"Lexa API error: {e.message}") print(f"Error code: {e.error_code}") except Exception as e: print(f"Unexpected error: {e}") ``` ```python Advanced Error Handling theme={null} from cerevox import Lexa, LexaError import time def robust_parse(client, files, max_retries=3): """Parse with custom retry logic""" for attempt in range(max_retries): try: return client.parse(files) except LexaError as e: if e.error_code == "RATE_LIMIT_EXCEEDED": wait_time = 2 ** attempt # Exponential backoff print(f"Rate limited, waiting {wait_time}s...") time.sleep(wait_time) continue else: raise e except Exception as e: if attempt == max_retries - 1: raise e print(f"Attempt {attempt + 1} failed, retrying...") time.sleep(1) raise Exception("Max retries exceeded") ``` ## Best Practices Use the async client for better performance when processing multiple files: ```python theme={null} import asyncio from cerevox import AsyncLexa async def process_documents(file_paths): async with AsyncLexa(api_key="your-api-key") as client: # Process multiple batches concurrently tasks = [] batch_size = 10 for i in range(0, len(file_paths), batch_size): batch = file_paths[i:i + batch_size] task = client.parse(batch) tasks.append(task) results = await asyncio.gather(*tasks) return [doc for batch in results for doc in batch] ``` Use progress callbacks for long-running operations: ```python theme={null} def progress_callback(status): print(f"Status: {status.status}") if hasattr(status, 'progress') and status.progress: print(f"Progress: {status.progress}") documents = client.parse( ["large-document.pdf"], progress_callback=progress_callback, timeout=300.0 # 5 minutes for large files ) ``` Properly manage client resources: ```python theme={null} # ✅ Good: Use context manager for async async with AsyncLexa(api_key="key") as client: documents = await client.parse(["file.pdf"]) # ✅ Good: Reuse sync client client = Lexa(api_key="key") for file_batch in file_batches: documents = client.parse(file_batch) process_documents(documents) # ❌ Avoid: Creating new clients repeatedly for file in files: client = Lexa(api_key="key") # Wasteful documents = client.parse([file]) ``` *** Ready to start parsing? Check out our [quickstart guide](/welcome/quickstart) or explore [real-world examples](/examples/basic-usage). # Configuration Source: https://docs.cerevox.ai/api/configuration Complete guide to configuring the Lexa client for optimal performance ## Client ### Basic Configuration ```python Basic Setup theme={null} from cerevox import Lexa # Minimal configuration client = Lexa(api_key="your-api-key") # Full configuration client = Lexa( api_key="your-api-key", base_url="https://data.cerevox.ai", timeout=120.0, max_retries=5, retry_delay=2.0, poll_interval=3.0, max_concurrent=10 ) ``` ```python Async Configuration theme={null} from cerevox import AsyncLexa # Async client with custom configuration async with AsyncLexa( api_key="your-api-key", timeout=300.0, max_retries=3, retry_delay=1.0, max_concurrent=20 ) as client: documents = await client.parse(["document.pdf"]) ``` ### Configuration Parameters Your Cerevox API key. Get one at [cerevox.ai/lexa](https://cerevox.ai/lexa) Base URL for the Cerevox API. Change for custom endpoints or testing Default timeout for API requests in seconds. Individual operations can override this Maximum number of automatic retry attempts for failed requests Base delay between retry attempts. Uses exponential backoff Default interval between job status checks during parsing Maximum number of concurrent requests (async client only) ## Environment Variables ### Standard Environment Variables ```bash Environment Setup theme={null} # Required export CEREVOX_API_KEY="your-api-key" # Optional Configuration export CEREVOX_BASE_URL="https://data.cerevox.ai" export CEREVOX_TIMEOUT="120" export CEREVOX_MAX_RETRIES="5" export CEREVOX_RETRY_DELAY="2.0" export CEREVOX_POLL_INTERVAL="3.0" export CEREVOX_MAX_CONCURRENT="20" # Logging Configuration export CEREVOX_LOG_LEVEL="INFO" export CEREVOX_LOG_FORMAT="json" ``` ```python Using Environment Variables theme={null} import os from cerevox import Lexa # Client automatically uses environment variables client = Lexa() # Override specific settings client = Lexa( timeout=float(os.getenv("CUSTOM_TIMEOUT", 60)), max_retries=int(os.getenv("CUSTOM_MAX_RETRIES", 3)) ) ``` ### Development vs Production ```bash Development Environment theme={null} # Development settings - more verbose, shorter timeouts export CEREVOX_API_KEY="dev-api-key" export CEREVOX_BASE_URL="https://dev.cerevox.ai" export CEREVOX_TIMEOUT="30" export CEREVOX_MAX_RETRIES="2" export CEREVOX_LOG_LEVEL="DEBUG" ``` ```bash Production Environment theme={null} # Production settings - optimized for reliability export CEREVOX_API_KEY="prod-api-key" export CEREVOX_BASE_URL="https://data.cerevox.ai" export CEREVOX_TIMEOUT="300" export CEREVOX_MAX_RETRIES="5" export CEREVOX_RETRY_DELAY="2.0" export CEREVOX_LOG_LEVEL="WARNING" export CEREVOX_MAX_CONCURRENT="50" ``` ## Performance Tuning ### Timeout Configuration ```python Dynamic Timeouts theme={null} import os from cerevox import Lexa def get_optimal_timeout(files): """Calculate optimal timeout based on file characteristics""" if isinstance(files, str): files = [files] total_size = 0 for file in files: if isinstance(file, str) and os.path.exists(file): total_size += os.path.getsize(file) # Base timeout + 1 second per MB base_timeout = 60 size_timeout = total_size / (1024 * 1024) # MB return base_timeout + size_timeout # Usage client = Lexa(api_key="your-api-key") files = ["large-document.pdf", "report.docx"] optimal_timeout = get_optimal_timeout(files) documents = client.parse( files, timeout=optimal_timeout, poll_interval=min(5.0, optimal_timeout / 20) ) ``` ```python Processing Mode Timeouts theme={null} from cerevox import Lexa, ProcessingMode # Different timeouts for different modes timeout_config = { ProcessingMode.DEFAULT: 120.0, # Accurate and fast ProcessingMode.ADVANCED: 600.0 # Even more accurate but slower } def parse_with_mode_timeout(client, files, mode): return client.parse( files, mode=mode, timeout=timeout_config.get(mode, 120.0) ) ``` ### Concurrency Configuration ```python Async Concurrency Limits theme={null} import asyncio from cerevox import AsyncLexa async def process_large_batch(files, batch_size=10, max_concurrent=5): """Process large batches with controlled concurrency""" # Configure client with concurrency limits async with AsyncLexa( api_key="your-api-key", max_concurrent=max_concurrent, timeout=300.0 ) as client: # Create semaphore to limit concurrent batches semaphore = asyncio.Semaphore(max_concurrent) async def process_batch(batch): async with semaphore: return await client.parse(batch) # Create batches batches = [files[i:i + batch_size] for i in range(0, len(files), batch_size)] # Process batches concurrently tasks = [process_batch(batch) for batch in batches] results = await asyncio.gather(*tasks, return_exceptions=True) # Handle results all_documents = [] for result in results: if isinstance(result, Exception): print(f"Batch failed: {result}") else: all_documents.extend(result) return all_documents ``` ```python Sync Batch Processing theme={null} from cerevox import Lexa from concurrent.futures import ThreadPoolExecutor, as_completed import time def process_batch_sync(files, max_workers=3): """Process files in parallel using threads""" client = Lexa( api_key="your-api-key", timeout=180.0, max_retries=3 ) def parse_single_file(file): try: return client.parse([file]) except Exception as e: print(f"Failed to parse {file}: {e}") return None all_documents = [] with ThreadPoolExecutor(max_workers=max_workers) as executor: # Submit all files future_to_file = { executor.submit(parse_single_file, file): file for file in files } # Collect results for future in as_completed(future_to_file): file = future_to_file[future] try: documents = future.result() if documents: all_documents.extend(documents) except Exception as e: print(f"Error processing {file}: {e}") return all_documents ``` ## Security ### API Key Management ```python Secure API Key Handling theme={null} import os from pathlib import Path from cerevox import Lexa class SecureLexaClient: def __init__(self): self.api_key = self._get_secure_api_key() self.client = Lexa(api_key=self.api_key) def _get_secure_api_key(self): """Get API key from secure sources""" # 1. Environment variable (recommended) api_key = os.getenv("CEREVOX_API_KEY") if api_key: return api_key # 2. Secure file (if environment variable not available) key_file = Path.home() / ".cerevox" / "api_key" if key_file.exists(): return key_file.read_text().strip() # 3. AWS Secrets Manager, Azure Key Vault, etc. # api_key = self._get_from_secrets_manager() raise ValueError("No API key found. Set CEREVOX_API_KEY environment variable.") def parse(self, *args, **kwargs): return self.client.parse(*args, **kwargs) # Usage client = SecureLexaClient() documents = client.parse(["document.pdf"]) ``` ```python Cloud Secret Management theme={null} import boto3 from cerevox import Lexa def get_api_key_from_aws_secrets(): """Get API key from AWS Secrets Manager""" secrets_client = boto3.client('secretsmanager') try: response = secrets_client.get_secret_value( SecretId='cerevox/api-key' ) return response['SecretString'] except Exception as e: raise ValueError(f"Failed to get API key from AWS Secrets: {e}") # Azure Key Vault example def get_api_key_from_azure(): """Get API key from Azure Key Vault""" from azure.keyvault.secrets import SecretClient from azure.identity import DefaultAzureCredential credential = DefaultAzureCredential() client = SecretClient( vault_url="https://your-vault.vault.azure.net/", credential=credential ) secret = client.get_secret("cerevox-api-key") return secret.value # Usage api_key = get_api_key_from_aws_secrets() client = Lexa(api_key=api_key) ``` ### Network Security ```python SSL/TLS Configuration theme={null} import ssl from cerevox import Lexa # Custom SSL context for enterprise environments ssl_context = ssl.create_default_context() ssl_context.check_hostname = True ssl_context.verify_mode = ssl.CERT_REQUIRED # For self-signed certificates (development only) # ssl_context.check_hostname = False # ssl_context.verify_mode = ssl.CERT_NONE client = Lexa( api_key="your-api-key", ssl_context=ssl_context ) ``` ```python Proxy Configuration theme={null} import os from cerevox import Lexa # Configure proxy settings proxies = { 'http': os.getenv('HTTP_PROXY'), 'https': os.getenv('HTTPS_PROXY') } client = Lexa( api_key="your-api-key", proxies=proxies, timeout=180.0 # Longer timeout for proxy connections ) ``` ## Logging ### Basic Logging Setup ```python Standard Logging theme={null} import logging from cerevox import Lexa # Configure logging logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s' ) # Enable Cerevox SDK logging cerevox_logger = logging.getLogger('cerevox') cerevox_logger.setLevel(logging.DEBUG) client = Lexa(api_key="your-api-key") documents = client.parse(["document.pdf"]) ``` ```python Structured Logging theme={null} import logging import json from datetime import datetime from cerevox import Lexa class JSONFormatter(logging.Formatter): def format(self, record): log_entry = { 'timestamp': datetime.utcnow().isoformat(), 'level': record.levelname, 'logger': record.name, 'message': record.getMessage(), 'module': record.module, 'function': record.funcName, 'line': record.lineno } # Add extra fields if hasattr(record, 'request_id'): log_entry['request_id'] = record.request_id if hasattr(record, 'error_code'): log_entry['error_code'] = record.error_code return json.dumps(log_entry) # Setup structured logging handler = logging.StreamHandler() handler.setFormatter(JSONFormatter()) logger = logging.getLogger('cerevox') logger.addHandler(handler) logger.setLevel(logging.INFO) ``` ### Production Logging ```python Production Logging Setup theme={null} import logging import logging.handlers from pathlib import Path from cerevox import Lexa def setup_production_logging(): """Setup production-grade logging""" # Create logs directory log_dir = Path("logs") log_dir.mkdir(exist_ok=True) # Configure root logger logging.basicConfig( level=logging.WARNING, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s' ) # Cerevox-specific logger with file rotation cerevox_logger = logging.getLogger('cerevox') cerevox_logger.setLevel(logging.INFO) # Rotating file handler file_handler = logging.handlers.RotatingFileHandler( log_dir / "cerevox.log", maxBytes=10*1024*1024, # 10MB backupCount=5 ) file_handler.setFormatter(logging.Formatter( '%(asctime)s - %(name)s - %(levelname)s - %(message)s' )) cerevox_logger.addHandler(file_handler) # Error-specific handler error_handler = logging.handlers.RotatingFileHandler( log_dir / "cerevox_errors.log", maxBytes=5*1024*1024, # 5MB backupCount=10 ) error_handler.setLevel(logging.ERROR) error_handler.setFormatter(logging.Formatter( '%(asctime)s - %(name)s - %(levelname)s - %(message)s\n%(exc_info)s' )) cerevox_logger.addHandler(error_handler) return cerevox_logger # Setup logging logger = setup_production_logging() client = Lexa(api_key="your-api-key") ``` ## Framework Integration ### Django ```python Django Settings theme={null} # settings.py import os # Cerevox configuration CEREVOX_API_KEY = os.getenv('CEREVOX_API_KEY') CEREVOX_TIMEOUT = 300.0 CEREVOX_MAX_RETRIES = 5 # Django logging configuration LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'verbose': { 'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}', 'style': '{', }, }, 'handlers': { 'cerevox_file': { 'level': 'INFO', 'class': 'logging.FileHandler', 'filename': 'logs/cerevox.log', 'formatter': 'verbose', }, }, 'loggers': { 'cerevox': { 'handlers': ['cerevox_file'], 'level': 'INFO', 'propagate': True, }, }, } ``` ```python Django Service Class theme={null} # services.py from django.conf import settings from cerevox import Lexa, LexaError import logging logger = logging.getLogger(__name__) class DocumentParsingService: def __init__(self): self.client = Lexa( api_key=settings.CEREVOX_API_KEY, timeout=settings.CEREVOX_TIMEOUT, max_retries=settings.CEREVOX_MAX_RETRIES ) def parse_documents(self, files): """Parse documents with Django integration""" try: documents = self.client.parse(files) logger.info(f"Successfully parsed {len(documents)} documents") return documents except LexaError as e: logger.error(f"Document parsing failed: {e.message}") raise ``` ### FastAPI ```python FastAPI Settings theme={null} # config.py from pydantic import BaseSettings class Settings(BaseSettings): cerevox_api_key: str cerevox_timeout: float = 120.0 cerevox_max_retries: int = 3 cerevox_base_url: str = "https://data.cerevox.ai" class Config: env_file = ".env" settings = Settings() ``` ```python FastAPI Dependency theme={null} # dependencies.py from fastapi import Depends from cerevox import AsyncLexa from .config import settings async def get_lexa_client(): """FastAPI dependency for Lexa client""" async with AsyncLexa( api_key=settings.cerevox_api_key, timeout=settings.cerevox_timeout, max_retries=settings.cerevox_max_retries ) as client: yield client # Usage in routes from fastapi import FastAPI, Depends, UploadFile from .dependencies import get_lexa_client app = FastAPI() @app.post("/parse") async def parse_documents( files: list[UploadFile], client: AsyncLexa = Depends(get_lexa_client) ): # Parse uploaded files file_contents = [await f.read() for f in files] documents = await client.parse(file_contents) return {"documents": len(documents)} ``` ## Validation ```python Configuration Validation theme={null} from cerevox import Lexa, LexaError import os from typing import Optional class ValidatedLexaConfig: def __init__( self, api_key: Optional[str] = None, base_url: Optional[str] = None, timeout: float = 60.0, max_retries: int = 3 ): self.api_key = self._validate_api_key(api_key) self.base_url = self._validate_base_url(base_url) self.timeout = self._validate_timeout(timeout) self.max_retries = self._validate_max_retries(max_retries) def _validate_api_key(self, api_key: Optional[str]) -> str: if not api_key: api_key = os.getenv('CEREVOX_API_KEY') if not api_key: raise ValueError("API key is required") if not api_key.startswith('cx_'): raise ValueError("Invalid API key format") return api_key def _validate_base_url(self, base_url: Optional[str]) -> str: if not base_url: base_url = os.getenv('CEREVOX_BASE_URL', 'https://data.cerevox.ai') if not base_url.startswith(('http://', 'https://')): raise ValueError("Base URL must start with http:// or https://") return base_url def _validate_timeout(self, timeout: float) -> float: if timeout <= 0: raise ValueError("Timeout must be positive") if timeout > 3600: # 1 hour max raise ValueError("Timeout too large (max 3600 seconds)") return timeout def _validate_max_retries(self, max_retries: int) -> int: if max_retries < 0: raise ValueError("Max retries cannot be negative") if max_retries > 10: raise ValueError("Max retries too large (max 10)") return max_retries def create_client(self) -> Lexa: """Create validated Lexa client""" return Lexa( api_key=self.api_key, base_url=self.base_url, timeout=self.timeout, max_retries=self.max_retries ) # Usage try: config = ValidatedLexaConfig( timeout=120.0, max_retries=5 ) client = config.create_client() print("Client configured successfully") except ValueError as e: print(f"Configuration error: {e}") ``` *** Explore [real-world examples](/examples/basic-usage) to see these configurations in action. # Error Handling Source: https://docs.cerevox.ai/api/error-handling Complete guide to handling errors and exceptions in Lexa ## Exception Hierarchy ```python Exception Types theme={null} from cerevox import ( LexaError, # Base exception LexaAuthError, # Authentication/authorization issues LexaRateLimitError, # Rate limiting LexaTimeoutError, # Request timeouts LexaJobFailedError, # Document processing failures LexaUnsupportedFileError, # Unsupported file types LexaValidationError, # Request validation errors LexaQuotaExceededError, # Usage quota exceeded LexaServerError # Server-side errors ) ``` ## Error Types ### LexaError (Base Exception) All Lexa-specific exceptions inherit from `LexaError`. ```python Basic Error Handling theme={null} from cerevox import Lexa, LexaError client = Lexa(api_key="your-api-key") try: documents = client.parse(["document.pdf"]) except LexaError as e: print(f"Lexa error: {e.message}") print(f"Status code: {e.status_code}") print(f"Request ID: {e.request_id}") print(f"Retry suggested: {e.retry_suggested}") ``` ```python LexaError Properties theme={null} # All LexaError instances have these properties: error.message # Human-readable error message error.status_code # HTTP status code (if applicable) error.request_id # Unique request identifier for debugging error.response_data # Additional error details (dict) error.retry_suggested # Whether retry is recommended (bool) ``` ### LexaAuthError Raised when API key authentication or authorization fails. ```python Common Causes theme={null} # Invalid API key client = Lexa(api_key="invalid-key") # Missing API key client = Lexa() # No CEREVOX_API_KEY environment variable # Expired or revoked API key # Access forbidden due to permissions ``` ```python Handling Authentication Errors theme={null} from cerevox import Lexa, LexaAuthError try: client = Lexa(api_key="your-api-key") documents = client.parse(["document.pdf"]) except LexaAuthError as e: print("Authentication failed!") print("Please check your API key at https://cerevox.ai/lexa") print(f"Error: {e.message}") print(f"Status: {e.status_code}") # 401 or 403 ``` **Status Codes:** * `401` - Invalid or expired API key * `403` - Access forbidden (insufficient permissions) ### LexaRateLimitError Raised when API rate limits are exceeded. ```python Rate Limit Handling theme={null} from cerevox import Lexa, LexaRateLimitError import time def parse_with_retry(client, files, max_retries=3): for attempt in range(max_retries): try: return client.parse(files) except LexaRateLimitError as e: if attempt == max_retries - 1: raise e # Use recommended retry delay wait_time = e.get_retry_delay() print(f"Rate limited. Waiting {wait_time} seconds...") time.sleep(wait_time) raise LexaRateLimitError("Max retries exceeded") ``` ```python Rate Limit Properties theme={null} try: documents = client.parse(["document.pdf"]) except LexaRateLimitError as e: print(f"Rate limit exceeded: {e.message}") print(f"Retry after: {e.retry_after} seconds") print(f"Recommended delay: {e.get_retry_delay()} seconds") print(f"Retry suggested: {e.retry_suggested}") # Always True ``` **Status Code:** `429` ### LexaValidationError Raised when request parameters are invalid. ```python Common Validation Errors theme={null} from cerevox import Lexa, LexaValidationError client = Lexa(api_key="your-api-key") try: # Invalid file path or parameters documents = client.parse(["/nonexistent/file.pdf"]) except LexaValidationError as e: print(f"Validation error: {e.message}") print(f"Validation errors: {e.validation_errors}") print(f"Retry suggested: {e.retry_suggested}") # Always False ``` ```python Validation Best Practices theme={null} import os from pathlib import Path def validate_files(files): """Validate files before parsing""" validated_files = [] for file in files: if isinstance(file, (str, Path)): file_path = Path(file) if not file_path.exists(): raise LexaValidationError(f"File not found: {file_path}") if file_path.stat().st_size == 0: raise LexaValidationError(f"File is empty: {file_path}") validated_files.append(str(file_path)) else: validated_files.append(file) return validated_files # Use validation try: validated_files = validate_files(["document.pdf", "report.docx"]) documents = client.parse(validated_files) except LexaValidationError as e: print(f"Validation failed: {e.message}") if e.validation_errors: for field, error in e.validation_errors.items(): print(f" {field}: {error}") ``` **Status Code:** `400` ### LexaJobFailedError Raised when document processing fails on the server side. ```python Processing Error Handling theme={null} from cerevox import Lexa, LexaJobFailedError try: documents = client.parse(["corrupted-document.pdf"]) except LexaJobFailedError as e: print(f"Processing failed: {e.message}") print(f"Job ID: {e.job_id}") print(f"Failure reason: {e.failure_reason}") print(f"Retry suggested: {e.retry_suggested}") ``` ```python Handling Specific Processing Errors theme={null} def handle_job_failure(error): """Handle different types of job failures""" if not error.retry_suggested: if "invalid_file_format" in (error.failure_reason or ""): return "File format not supported. Check supported formats." elif "file_corrupted" in (error.failure_reason or ""): return "File appears corrupted. Please check the source file." elif "file_too_large" in (error.failure_reason or ""): return "File exceeds size limits. Try splitting the file." else: return f"Processing failed: {error.message}" else: return "Temporary processing failure. Retry recommended." try: documents = client.parse(["document.pdf"]) except LexaJobFailedError as e: advice = handle_job_failure(e) print(advice) if e.retry_suggested: print("You can retry this request.") ``` **Common Failure Reasons:** * `invalid_file_format` - Unsupported file format (not retryable) * `file_corrupted` - File is corrupted (not retryable) * `file_too_large` - File exceeds size limits (not retryable) * `unsupported_format` - Format not supported (not retryable) * `temporary_server_error` - Temporary issue (retryable) ### LexaUnsupportedFileError Raised when attempting to process unsupported file formats. ```python Unsupported File Handling theme={null} from cerevox import Lexa, LexaUnsupportedFileError try: documents = client.parse(["document.xyz"]) except LexaUnsupportedFileError as e: print(f"Unsupported file type: {e.message}") print(f"File type: {e.file_type}") print(f"Supported types: {e.supported_types}") print(f"Retry suggested: {e.retry_suggested}") # Always False ``` ```python File Type Validation theme={null} def check_file_support(file_path, supported_types=None): """Check if file type is supported""" if supported_types is None: supported_types = ['.pdf', '.docx', '.txt', '.md'] # Example file_ext = Path(file_path).suffix.lower() if file_ext not in supported_types: raise LexaUnsupportedFileError( f"File type {file_ext} not supported", file_type=file_ext, supported_types=supported_types ) return True # Usage try: check_file_support("document.pdf") documents = client.parse(["document.pdf"]) except LexaUnsupportedFileError as e: print(f"Cannot process file: {e.message}") ``` **Status Code:** `415` ### LexaTimeoutError Raised when operations exceed timeout limits. ```python Timeout Handling theme={null} from cerevox import Lexa, LexaTimeoutError import time def parse_with_custom_timeout(client, files, base_timeout=60): """Parse with dynamic timeout based on file size""" # Calculate timeout based on file sizes total_size = sum(os.path.getsize(f) for f in files if isinstance(f, str)) timeout = base_timeout + (total_size // (1024 * 1024)) # +1s per MB try: return client.parse(files, timeout=timeout) except LexaTimeoutError as e: print(f"Operation timed out after {e.timeout_duration} seconds") print(f"Retry suggested: {e.retry_suggested}") # True print("Try with longer timeout or split the files") raise ``` ```python Async Timeout Handling theme={null} import asyncio from cerevox import AsyncLexa, LexaTimeoutError async def parse_with_cancellation(files): """Parse with cancellation support""" async with AsyncLexa(api_key="your-api-key") as client: try: # Race between parsing and timeout documents = await asyncio.wait_for( client.parse(files), timeout=300.0 ) return documents except asyncio.TimeoutError: print("Operation cancelled due to timeout") raise LexaTimeoutError("Async operation timed out") ``` **Status Code:** `408` ### LexaQuotaExceededError Raised when usage quotas are exceeded. ```python Quota Handling theme={null} from cerevox import Lexa, LexaQuotaExceededError try: documents = client.parse(["document.pdf"]) except LexaQuotaExceededError as e: print(f"Quota exceeded: {e.message}") print(f"Quota type: {e.quota_type}") print(f"Reset time: {e.reset_time}") print(f"Retry suggested: {e.retry_suggested}") # True if reset_time exists if e.reset_time: print(f"Quota resets at: {e.reset_time}") else: print("Contact support to increase quota") ``` ```python Quota Management theme={null} def handle_quota_exceeded(error): """Handle quota exceeded errors""" if error.quota_type == "monthly": return "Monthly quota exceeded. Upgrade plan or wait for reset." elif error.quota_type == "daily": return "Daily quota exceeded. Try again tomorrow." elif error.quota_type == "concurrent": return "Too many concurrent requests. Reduce parallelism." else: return f"Quota exceeded: {error.message}" try: documents = client.parse(["document.pdf"]) except LexaQuotaExceededError as e: advice = handle_quota_exceeded(e) print(advice) ``` **Status Code:** `402` ### LexaServerError Raised for server-side errors (5xx status codes). ```python Server Error Handling theme={null} from cerevox import Lexa, LexaServerError import time def robust_parse(client, files, max_retries=5): """Parse with server error recovery""" for attempt in range(max_retries): try: return client.parse(files) except LexaServerError as e: if attempt == max_retries - 1: raise e # Exponential backoff for server errors wait_time = 2 ** attempt print(f"Server error on attempt {attempt + 1}: {e.message}") print(f"Retrying in {wait_time} seconds...") time.sleep(wait_time) raise LexaServerError("Max server error retries exceeded") ``` **Status Codes:** `500`, `502`, `503`, `504` ## Intelligent Error Classification The SDK provides intelligent error classification through the `create_error_from_response` function. ```python Error Classification theme={null} from cerevox.exceptions import create_error_from_response # Automatically classifies errors based on: # - HTTP status code # - Response error_type field # - Error message content # - Additional response metadata def handle_api_response(status_code, response_data, request_id=None): """Example of automatic error classification""" if status_code >= 400: error = create_error_from_response(status_code, response_data, request_id) # Error is automatically classified to the correct type if isinstance(error, LexaRateLimitError): print(f"Rate limited. Retry after {error.get_retry_delay()}s") elif isinstance(error, LexaAuthError): print("Authentication failed. Check API key.") elif isinstance(error, LexaJobFailedError): print(f"Job {error.job_id} failed: {error.failure_reason}") raise error ``` ## Retry Strategies The SDK provides intelligent retry guidance through the `get_retry_strategy` function. ```python Intelligent Retry theme={null} from cerevox.exceptions import get_retry_strategy import time def smart_retry_parse(client, files): """Parse with intelligent retry strategy""" max_attempts = 3 for attempt in range(max_attempts): try: return client.parse(files) except LexaError as e: strategy = get_retry_strategy(e) if not strategy["should_retry"] or attempt == max_attempts - 1: print(f"Not retrying: {strategy['reason']}") raise e delay = strategy["delay"] backoff = strategy["backoff"] # Apply backoff strategy if backoff == "exponential": delay = delay * (2 ** attempt) elif backoff == "linear": delay = delay * (attempt + 1) # "fixed" backoff uses delay as-is print(f"Retrying in {delay}s ({strategy['reason']})") time.sleep(delay) raise RuntimeError("Max retries exceeded") ``` ```python Retry Strategy Examples theme={null} # Different errors get different retry strategies: # Rate limit error: # { # "should_retry": True, # "delay": 60, # From retry_after or default # "backoff": "fixed", # "max_retries": 3, # "reason": "Rate limit - use fixed delay" # } # Server error: # { # "should_retry": True, # "delay": 2, # "backoff": "exponential", # "max_retries": 5, # "reason": "Server error - aggressive retry" # } # Validation error: # { # "should_retry": False, # "reason": "Error type not suitable for retry", # "delay": 0, # "max_retries": 0 # } ``` ## Best Practices ### Comprehensive Error Handling ```python Production Error Handling theme={null} from cerevox import Lexa, LexaError, LexaAuthError, LexaRateLimitError from cerevox.exceptions import get_retry_strategy import logging import time # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) class RobustLexaClient: def __init__(self, api_key, max_retries=3): self.client = Lexa(api_key=api_key) self.max_retries = max_retries def parse_documents(self, files, mode=None): """Parse documents with comprehensive error handling""" for attempt in range(self.max_retries): try: return self.client.parse(files, mode=mode) except LexaError as e: strategy = get_retry_strategy(e) # Log the error logger.error( f"Parsing failed (attempt {attempt + 1}): {e.message}", extra={ 'error_type': type(e).__name__, 'status_code': e.status_code, 'request_id': e.request_id, 'retry_suggested': e.retry_suggested } ) # Don't retry certain error types if isinstance(e, (LexaAuthError, LexaValidationError, LexaUnsupportedFileError)): logger.error(f"Non-retryable error: {e.message}") raise e # Check if we should retry if not strategy["should_retry"] or attempt == self.max_retries - 1: logger.error(f"Not retrying: {strategy['reason']}") raise e # Calculate delay with backoff delay = strategy["delay"] if strategy["backoff"] == "exponential": delay = delay * (2 ** attempt) elif strategy["backoff"] == "linear": delay = delay * (attempt + 1) logger.warning(f"Retrying in {delay}s ({strategy['reason']})") time.sleep(delay) raise RuntimeError("Max retries exceeded") # Usage client = RobustLexaClient(api_key="your-api-key") try: documents = client.parse_documents(["document.pdf"]) print(f"Successfully parsed {len(documents)} documents") except LexaError as e: print(f"Parsing failed: {e.message}") ``` ### Error Monitoring and Logging ```python Error Monitoring theme={null} import logging from cerevox import Lexa, LexaError # Custom error handler class LexaErrorHandler(logging.Handler): def emit(self, record): if record.name == 'cerevox' and record.levelno >= logging.ERROR: # Send to monitoring service self.send_to_monitoring(record) def send_to_monitoring(self, record): # Implementation depends on your monitoring solution pass # Configure logging logger = logging.getLogger('cerevox') logger.addHandler(LexaErrorHandler()) def monitored_parse(client, files): """Parse with error monitoring""" try: return client.parse(files) except LexaError as e: logger.error( f"Parsing failed", extra={ 'error_type': type(e).__name__, 'status_code': e.status_code, 'request_id': e.request_id, 'files': len(files) if isinstance(files, list) else 1, 'message': e.message, 'retry_suggested': e.retry_suggested } ) raise ``` ### Graceful Degradation ```python Fallback Strategies theme={null} from cerevox import Lexa, ProcessingMode, LexaError, LexaJobFailedError def parse_with_fallback(client, files): """Parse with fallback to simpler processing modes""" # Try processing modes in order of preference modes = [ProcessingMode.ADVANCED, ProcessingMode.DEFAULT] for mode in modes: try: return client.parse(files, mode=mode) except LexaJobFailedError as e: if not e.retry_suggested and mode != ProcessingMode.DEFAULT: print(f"Failed with {mode.value} mode, trying simpler mode...") continue raise e except LexaError as e: if mode == ProcessingMode.DEFAULT: raise e # All modes failed continue raise RuntimeError("All processing modes failed") def parse_with_chunking(client, files, chunk_size=10): """Parse large batches in smaller chunks""" all_documents = [] for i in range(0, len(files), chunk_size): chunk = files[i:i + chunk_size] try: documents = client.parse(chunk) all_documents.extend(documents) except LexaError as e: print(f"Chunk {i//chunk_size + 1} failed: {e.message}") # Continue with remaining chunks if error allows if not e.retry_suggested: continue raise e return all_documents ``` ## Error Recovery Patterns ### Circuit Breaker Pattern ```python Circuit Breaker theme={null} import time from enum import Enum class CircuitState(Enum): CLOSED = "closed" OPEN = "open" HALF_OPEN = "half_open" class CircuitBreaker: def __init__(self, failure_threshold=5, recovery_timeout=60): self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.failure_count = 0 self.last_failure_time = None self.state = CircuitState.CLOSED def call(self, func, *args, **kwargs): if self.state == CircuitState.OPEN: if time.time() - self.last_failure_time > self.recovery_timeout: self.state = CircuitState.HALF_OPEN else: raise RuntimeError("Circuit breaker is OPEN") try: result = func(*args, **kwargs) self.on_success() return result except Exception as e: self.on_failure() raise e def on_success(self): self.failure_count = 0 self.state = CircuitState.CLOSED def on_failure(self): self.failure_count += 1 self.last_failure_time = time.time() if self.failure_count >= self.failure_threshold: self.state = CircuitState.OPEN # Usage circuit_breaker = CircuitBreaker() client = Lexa(api_key="your-api-key") try: documents = circuit_breaker.call(client.parse, ["document.pdf"]) except RuntimeError as e: print(f"Circuit breaker active: {e}") ``` *** Learn about [client configuration](/api/configuration) for optimal performance and reliability. # API Methods Source: https://docs.cerevox.ai/api/methods Complete reference for all Lexa API methods and operations ## Core Parsing Methods ### parse() Parse local files, file-like objects, or raw bytes content. ```python Signature theme={null} def parse( files: Union[str, Path, bytes, BinaryIO, List[Union[str, Path, bytes, BinaryIO]]], mode: ProcessingMode = ProcessingMode.DEFAULT, progress_callback: Optional[Callable[[JobStatus], None]] = None, timeout: float = 60.0, poll_interval: float = 2.0 ) -> DocumentBatch ``` ```python Usage Examples theme={null} from cerevox import Lexa, ProcessingMode client = Lexa(api_key="your-api-key") # Parse single file documents = client.parse("document.pdf") # Parse multiple files documents = client.parse(["doc1.pdf", "doc2.docx", "doc3.txt"]) # Parse with custom settings documents = client.parse( files=["document.pdf"], mode=ProcessingMode.ADVANCED, timeout=120.0, poll_interval=1.0 ) # Parse bytes content with open("document.pdf", "rb") as f: content = f.read() documents = client.parse(content) # Parse with progress callback def progress_callback(status): print(f"Status: {status.status}") documents = client.parse( ["large-document.pdf"], progress_callback=progress_callback ) ``` **Parameters:** Files to parse. Can be: * File path (string or Path object) * Raw bytes content * File-like object (BinaryIO) * List of any of the above Processing mode: `DEFAULT` or `ADVANCED` Callback function to monitor parsing progress Maximum time to wait for parsing completion (seconds) Interval between status checks (seconds) **Returns:** `DocumentBatch` - Collection of parsed documents *** ### parse\_urls() Parse documents from URLs. ```python Signature theme={null} def parse_urls( urls: Union[str, List[str]], mode: ProcessingMode = ProcessingMode.DEFAULT, progress_callback: Optional[Callable[[JobStatus], None]] = None, timeout: float = 120.0, poll_interval: float = 2.0 ) -> DocumentBatch ``` ```python Usage Examples theme={null} # Parse single URL documents = client.parse_urls("https://example.com/document.pdf") # Parse multiple URLs urls = [ "https://example.com/doc1.pdf", "https://example.com/doc2.docx" ] documents = client.parse_urls(urls) # Parse with custom settings documents = client.parse_urls( urls=["https://example.com/large-document.pdf"], mode=ProcessingMode.ADVANCED, timeout=300.0, poll_interval=5.0 ) ``` **Parameters:** URLs to parse. Can be a single URL string or list of URLs Processing mode: `DEFAULT` or `ADVANCED` Callback function to monitor parsing progress Maximum time to wait for parsing completion (seconds) Interval between status checks (seconds) **Returns:** `DocumentBatch` - Collection of parsed documents *** ### get\_job\_status() Get the current status of a parsing job. ```python Signature theme={null} def get_job_status(job_id: str) -> JobStatus ``` ```python Usage Examples theme={null} # Get job status status = client.get_job_status("job_123456") print(f"Status: {status.status}") print(f"Progress: {status.progress}") # Monitor job until completion import time while True: status = client.get_job_status("job_123456") print(f"Current status: {status.status}") if status.status in ["COMPLETED", "FAILED"]: break time.sleep(2) ``` **Parameters:** The job ID to check status for **Returns:** `JobStatus` - Current job status information *** ## Amazon S3 Methods ### list\_s3\_buckets() List available S3 buckets. ```python Signature theme={null} def list_s3_buckets() -> S3BucketList ``` ```python Usage Examples theme={null} # List all buckets buckets = client.list_s3_buckets() print(f"Found {len(buckets.buckets)} buckets") for bucket in buckets.buckets: print(f"Bucket: {bucket.name}") print(f"Created: {bucket.creation_date}") ``` **Returns:** `S3BucketList` - List of available S3 buckets *** ### list\_s3\_folder() List contents of an S3 folder. ```python Signature theme={null} def list_s3_folder( bucket: str, folder_path: str = "", max_items: int = 1000 ) -> S3FolderContents ``` ```python Usage Examples theme={null} # List root folder contents = client.list_s3_folder("my-bucket") # List specific folder contents = client.list_s3_folder("my-bucket", "documents/") # List with custom limit contents = client.list_s3_folder( bucket="my-bucket", folder_path="documents/", max_items=100 ) # Display contents for item in contents.files: print(f"File: {item.key} ({item.size} bytes)") ``` **Parameters:** S3 bucket name Path within the bucket (empty for root) Maximum number of items to return **Returns:** `S3FolderContents` - Contents of the S3 folder *** ### parse\_s3\_folder() Parse all documents in an S3 folder. ```python Signature theme={null} def parse_s3_folder( bucket: str, folder_path: str = "", mode: ProcessingMode = ProcessingMode.DEFAULT, progress_callback: Optional[Callable[[JobStatus], None]] = None, timeout: float = 300.0, poll_interval: float = 5.0 ) -> DocumentBatch ``` ```python Usage Examples theme={null} # Parse entire bucket documents = client.parse_s3_folder("my-bucket") # Parse specific folder documents = client.parse_s3_folder("my-bucket", "documents/") # Parse with progress monitoring def progress_callback(status): print(f"Progress: {status.progress}") documents = client.parse_s3_folder( bucket="my-bucket", folder_path="documents/", progress_callback=progress_callback, timeout=600.0 ) ``` **Parameters:** S3 bucket name Path within the bucket to parse Processing mode: `DEFAULT` or `ADVANCED` Callback function to monitor parsing progress Maximum time to wait for parsing completion (seconds) Interval between status checks (seconds) **Returns:** `DocumentBatch` - Collection of parsed documents *** ## Microsoft SharePoint Methods ### list\_sharepoint\_sites() List available SharePoint sites. ```python Signature theme={null} def list_sharepoint_sites() -> SharePointSiteList ``` ```python Usage Examples theme={null} # List all sites sites = client.list_sharepoint_sites() for site in sites.sites: print(f"Site: {site.name} (ID: {site.id})") print(f"URL: {site.web_url}") ``` **Returns:** `SharePointSiteList` - List of available SharePoint sites *** ### list\_sharepoint\_drives() List drives in a SharePoint site. ```python Signature theme={null} def list_sharepoint_drives(site_id: str) -> SharePointDriveList ``` ```python Usage Examples theme={null} # List drives in a site drives = client.list_sharepoint_drives("site-id-123") for drive in drives.drives: print(f"Drive: {drive.name} (ID: {drive.id})") print(f"Type: {drive.drive_type}") ``` **Parameters:** SharePoint site ID **Returns:** `SharePointDriveList` - List of drives in the site *** ### parse\_sharepoint\_folder() Parse documents in a SharePoint folder. ```python Signature theme={null} def parse_sharepoint_folder( site_id: str, drive_id: str, folder_path: str = "", mode: ProcessingMode = ProcessingMode.DEFAULT, progress_callback: Optional[Callable[[JobStatus], None]] = None, timeout: float = 300.0, poll_interval: float = 5.0 ) -> DocumentBatch ``` ```python Usage Examples theme={null} # Parse SharePoint folder documents = client.parse_sharepoint_folder( site_id="site-123", drive_id="drive-456", folder_path="Documents/" ) # Parse with progress monitoring documents = client.parse_sharepoint_folder( site_id="site-123", drive_id="drive-456", folder_path="Documents/", progress_callback=lambda status: print(f"Progress: {status.progress}"), mode=ProcessingMode.ADVANCED, timeout=600.0 ) ``` **Parameters:** SharePoint site ID SharePoint drive ID Path within the drive to parse Processing mode: `DEFAULT` or `ADVANCED` Callback function to monitor parsing progress Maximum time to wait for parsing completion (seconds) Interval between status checks (seconds) **Returns:** `DocumentBatch` - Collection of parsed documents *** ## Box Methods ### list\_box\_folders() List folders in Box. ```python Signature theme={null} def list_box_folders(parent_folder_id: str = "0") -> BoxFolderList ``` ```python Usage Examples theme={null} # List root folders folders = client.list_box_folders() # List specific folder contents folders = client.list_box_folders("123456789") for folder in folders.folders: print(f"Folder: {folder.name} (ID: {folder.id})") ``` **Parameters:** Parent folder ID ("0" for root folder) **Returns:** `BoxFolderList` - List of folders *** ### parse\_box\_folder() Parse documents in a Box folder. ```python Signature theme={null} def parse_box_folder( folder_id: str, mode: ProcessingMode = ProcessingMode.DEFAULT, progress_callback: Optional[Callable[[JobStatus], None]] = None, timeout: float = 300.0, poll_interval: float = 5.0 ) -> DocumentBatch ``` ```python Usage Examples theme={null} # Parse Box folder documents = client.parse_box_folder("123456789") # Parse with custom settings documents = client.parse_box_folder( folder_id="123456789", mode=ProcessingMode.ADVANCED, timeout=600.0 ) ``` **Parameters:** Box folder ID to parse Processing mode: `DEFAULT` or `ADVANCED` Callback function to monitor parsing progress Maximum time to wait for parsing completion (seconds) Interval between status checks (seconds) **Returns:** `DocumentBatch` - Collection of parsed documents *** ## Dropbox Methods ### list\_dropbox\_folders() List folders in Dropbox. ```python Signature theme={null} def list_dropbox_folders(folder_path: str = "") -> DropboxFolderList ``` ```python Usage Examples theme={null} # List root folders folders = client.list_dropbox_folders() # List specific folder folders = client.list_dropbox_folders("/Documents") for folder in folders.folders: print(f"Folder: {folder.name}") print(f"Path: {folder.path_display}") ``` **Parameters:** Dropbox folder path (empty for root) **Returns:** `DropboxFolderList` - List of folders *** ### parse\_dropbox\_folder() Parse documents in a Dropbox folder. ```python Signature theme={null} def parse_dropbox_folder( folder_path: str, mode: ProcessingMode = ProcessingMode.DEFAULT, progress_callback: Optional[Callable[[JobStatus], None]] = None, timeout: float = 300.0, poll_interval: float = 5.0 ) -> DocumentBatch ``` ```python Usage Examples theme={null} # Parse Dropbox folder documents = client.parse_dropbox_folder("/Documents") # Parse with progress monitoring documents = client.parse_dropbox_folder( folder_path="/Documents", progress_callback=lambda status: print(f"Status: {status.status}"), mode=ProcessingMode.ADVANCED, timeout=600.0 ) ``` **Parameters:** Dropbox folder path to parse Processing mode: `DEFAULT` or `ADVANCED` Callback function to monitor parsing progress Maximum time to wait for parsing completion (seconds) Interval between status checks (seconds) **Returns:** `DocumentBatch` - Collection of parsed documents *** ## Async Methods All methods are available in async versions with the `AsyncLexa` client: ```python Async Usage theme={null} import asyncio from cerevox import AsyncLexa, ProcessingMode async def main(): async with AsyncLexa(api_key="your-api-key") as client: # All methods are available with await documents = await client.parse(["document.pdf"]) # Concurrent processing tasks = [ client.parse(["doc1.pdf"]), client.parse(["doc2.pdf"]), client.parse_urls(["https://example.com/doc.pdf"]) ] results = await asyncio.gather(*tasks) all_documents = [doc for batch in results for doc in batch] return all_documents asyncio.run(main()) ``` *** Learn about [error handling](/api/error-handling) and [client configuration](/api/configuration) for production use. # About Source: https://docs.cerevox.ai/company/about # The AI-Powered Data Layer for Developers 🧠⚡ At Cerevox, we've built **Lexa** - the enterprise-grade document parsing API that developers trust to extract structured data from any document. ## The Developer Challenge Modern applications demand intelligent document processing capabilities, but existing solutions force developers to choose between accuracy and performance. Traditional document parsing libraries struggle with: * **Complex layouts** and multi-format documents * **Poor accuracy** with tables, images, and structured data * **Slow processing** that doesn't scale with enterprise needs * **Limited integrations** with modern vector databases and AI workflows * **Inconsistent results** across different document types ## Our Solution: Lexa API **Lexa** is our state-of-the-art document parsing API that delivers: ### 🚀 **10x Performance & Enterprise Scale** * **Native async support** with concurrent processing * **Sub-second parsing** for most document types * **Automatic retries** with enterprise-grade reliability * **99.9% uptime SLA** for production workloads ### 🧠 **SOTA AI-Powered Extraction** * **Highest accuracy** in the industry using cutting-edge ML models * **Advanced table extraction** preserving structure and formatting * **12+ file formats** including PDF, DOCX, PPTX, HTML, and more * **Smart content chunking** optimized for RAG applications ### 🔗 **Vector Database Ready** * **Pre-optimized chunks** for embedding models * **Rich metadata extraction** including images, formatting, and structure * **Built-in integrations** with Pinecone, Weaviate, ChromaDB, and Qdrant * **Seamless RAG workflow** integration ### ☁️ **Cloud-Native Architecture** * **7+ cloud storage** integrations (S3, SharePoint, Google Drive, Box, Dropbox) * **REST API** with comprehensive Python SDK * **Framework agnostic** - works with Django, Flask, FastAPI * **Kubernetes-ready** with horizontal scaling ## Developer-First Experience ```python theme={null} # Get started in seconds pip install cerevox # Parse any document with 3 lines of code from cerevox import Lexa client = Lexa(api_key="your-api-key") documents = client.parse(["report.pdf", "data.xlsx"]) # Vector DB ready chunks chunks = documents.get_all_text_chunks(target_size=500) ``` ## Real-World Impact **Fortune 500 companies** use Cerevox to: * **Process millions** of documents daily with 99.9% accuracy * **Build RAG applications** that understand complex enterprise documents * **Extract structured data** from financial reports, legal contracts, and research papers * **Automate workflows** that previously required manual data entry ## What Makes Us Different * **🎯 Accuracy**: Industry-leading accuracy with state-of-the-art AI models * **⚡ Speed**: 10x faster than traditional parsing solutions * **🔧 Developer UX**: Simple API, comprehensive SDK, extensive documentation * **🏢 Enterprise Ready**: SOC2 compliant, 99.9% SLA, dedicated support * **🤖 AI-Optimized**: Built specifically for modern AI and RAG workflows ## Looking Forward We're expanding our platform with: * **Real-time document processing** pipelines * **Advanced AI extraction** for specialized document types * **Enhanced integrations** with popular developer tools * **Multi-modal capabilities** including image and video processing * **On-premise deployment** options for enterprise security *** **Ready to transform your document processing?** Join thousands of developers building the future with Cerevox. [Get Started with Lexa →](https://cerevox.ai/lexa) | [View API Docs →](https://docs.cerevox.ai) | [Join our Discord →](https://discord.gg/cerevox) # FAQ Source: https://docs.cerevox.ai/company/faq Frequently Asked Questions ## Getting Started Lexa is Cerevox's enterprise-grade document parsing API that delivers 10x better performance and accuracy compared to traditional solutions. Unlike other APIs that struggle with complex layouts and structured data, Lexa uses state-of-the-art AI models to extract content with 99.9% accuracy while maintaining native async support and vector database optimization. Key differentiators: * **SOTA accuracy** with advanced ML models * **10x faster** processing with sub-second response times * **Vector DB ready** chunks optimized for RAG applications * **12+ file formats** with consistent results * **Enterprise-grade** reliability with 99.9% SLA You can start parsing documents in under 5 minutes: ```python theme={null} pip install cerevox ``` ```python theme={null} from cerevox import Lexa client = Lexa(api_key="your-api-key") documents = client.parse(["document.pdf"]) # Get vector DB ready chunks chunks = documents.get_all_text_chunks(target_size=500) ``` Our Python SDK handles authentication, retries, and error handling automatically. We also provide comprehensive examples for Django, Flask, FastAPI, and async applications. Lexa supports 12+ file formats with consistent, high-accuracy parsing: **Documents**: PDF, DOCX, PPTX, TXT, HTML, RTF\ **Spreadsheets**: XLSX, CSV, TSV\ **Google Workspace**: Google Docs, Sheets, Slides\ **Data**: JSON, Parquet All formats support advanced table extraction, image detection, and metadata preservation. File size limits range from 100MB for complex documents to 1GB+ for simple text files. ## Technical Implementation Lexa provides native async support with the `AsyncLexa` client: ```python theme={null} import asyncio from cerevox import AsyncLexa async def main(): async with AsyncLexa(api_key="your-api-key") as client: # Process multiple documents concurrently documents = await client.parse([ "report1.pdf", "report2.docx", "data.xlsx" ]) # Batch process with progress tracking async for status in client.parse_with_progress(files): print(f"Progress: {status.progress}") asyncio.run(main()) ``` This enables concurrent processing of multiple documents, significantly improving throughput for batch operations. Lexa is designed specifically for RAG workflows with built-in vector database optimization: ```python theme={null} # Get optimally sized chunks for embeddings chunks = documents.get_all_text_chunks( target_size=500, # Optimal for most embedding models overlap=50, # Maintain context between chunks include_metadata=True # Rich metadata for filtering ) # Direct integration with vector databases for chunk in chunks: # Each chunk includes source document, page numbers, # element types, and confidence scores vector_db.upsert( id=chunk.id, vector=embed(chunk.content), metadata=chunk.metadata ) ``` We provide pre-built integration examples for Pinecone, Weaviate, ChromaDB, and Qdrant. Lexa integrates with 7+ major cloud storage platforms: * **Amazon S3**: Direct parsing from S3 buckets and folders * **Microsoft SharePoint**: Sites, drives, and document libraries * **Google Drive**: Files and folders with permission management * **Box**: Enterprise file storage with advanced metadata * **Dropbox**: Personal and business accounts * **Salesforce**: Document attachments and files * **Coming Soon**: Azure Blob, OneDrive, Notion ```python theme={null} # Parse entire S3 folder documents = client.parse_s3_folder( bucket="my-bucket", folder="documents/", recursive=True ) ``` ## Performance and Scaling Lexa delivers industry-leading performance across all metrics: **Speed**: * Simple PDFs: \< 1 second * Complex documents (100+ pages): 15-45 seconds * Batch processing: 10-50 documents/minute * Concurrent async: 100+ documents/minute **Accuracy**: * Text extraction: 99.9% * Table structure: 92.5% * Metadata extraction: 99.2% * Multi-format consistency: 99.7% **Reliability**: * API uptime: 99.9% SLA * Auto-retry on failures: 3 attempts with exponential backoff * Rate limiting: 1000 requests/minute (enterprise plans) Lexa is built for enterprise scale with several optimization strategies: **Horizontal Scaling**: Our API automatically scales to handle spikes in demand\ **Batch Processing**: Process up to 100 documents per API call\ **Async Processing**: Non-blocking operations with progress callbacks\ **Caching**: Intelligent caching reduces processing time for similar documents\ **Load Balancing**: Global infrastructure ensures low latency worldwide ```python theme={null} # Batch processing example large_batch = ["doc1.pdf", "doc2.docx", ...] # 100+ files documents = await client.parse( large_batch, progress_callback=lambda status: print(f"Progress: {status.completed}/{status.total}") ) ``` See [pricing](https://cerevox.ai/pricing) **Free Plan** (Free): * 1000 Documents Parsed * Community support **Dev Plan** (\$5/month): * Start with 100 pages * \$0.05 per additional page * 100 requests/minute * Email support * Vector DB integrations **Pro Plan** (\$99/month): * Start with 10,000 pages * \$0.01 per additional page * 3x cost for advanced processing * 100 requests/minute * Email support * Vector DB integrations **Enterprise Plan** (Custom): * Unlimited pages * 1000+ requests/minute * Dedicated support * On-premise deployment * Custom integrations All plans include the same high accuracy and all file format support. ## Advanced Features Lexa uses advanced computer vision and ML models to extract tables with high fidelity: ```python theme={null} documents = client.parse("financial_report.pdf") # Access extracted tables for doc in documents: for table in doc.tables: print(f"Table on page {table.page_number}") print(f"Dimensions: {table.rows}x{table.columns}") # Export to pandas DataFrame df = table.to_pandas() # Or get raw structured data data = table.to_dict() ``` Features include: * **Structure preservation**: Maintains cell relationships and formatting * **Multi-page tables**: Automatically combines split tables * **Header detection**: Identifies and preserves table headers * **Data type inference**: Automatically detects numbers, dates, etc. Lexa offers several processing modes and customization options: ```python theme={null} from cerevox import ProcessingMode # Standard processing (fastest) docs = client.parse("document.pdf", mode=ProcessingMode.DEFAULT) # Advanced processing (highest accuracy) docs = client.parse("document.pdf", mode=ProcessingMode.ADVANCED) # Custom chunking parameters chunks = docs.get_text_chunks( target_size=1000, # Larger chunks for long-form content tolerance=0.2, # 20% size variance allowed respect_boundaries=True, # Don't break sentences/paragraphs include_tables=True # Include table content in chunks ) ``` Contact our team for specialized processing modes for specific document types or industries. Security is built into every aspect of Lexa: **Data Security**: * TLS 1.3 encryption for all API communications * Documents processed in isolated environments * No document storage - processed and deleted immediately * SOC 2 Type II certified infrastructure **Access Control**: * API key authentication with rotation support * Role-based access control (enterprise plans) * IP whitelisting and VPC connectivity options * Audit logging for all API operations **Compliance**: * GDPR compliant data processing * HIPAA compliance available (enterprise) * Regional data processing options (US, EU, Asia) * On-premise deployment for maximum security ## Support and Community We provide comprehensive support across multiple channels: **Community Support**: * [Discord Community](https://discord.gg/cerevox) - Real-time chat with developers * [GitHub Discussions](https://github.com/CerevoxAI/cerevox-python/discussions) - Technical discussions * [Stack Overflow](https://stackoverflow.com/questions/tagged/cerevox) - Q\&A with the community **Direct Support**: * Email support for Pro and Enterprise customers * Video calls for Enterprise customers * Dedicated Slack channels for large deployments * 24/7 support for mission-critical applications Stay connected with the Cerevox developer community: * **[GitHub Repository](https://github.com/CerevoxAI/cerevox-python)**: Star for updates and releases * **[Discord Community](https://discord.gg/cerevox)**: Join 1000+ developers * **[Documentation](https://docs.cerevox.ai)**: Always up-to-date guides and examples We ship new features and improvements every 2-3 weeks based on community feedback. # Advanced Patterns Source: https://docs.cerevox.ai/examples/advanced-patterns Production-ready patterns for complex document processing workflows **Ready for production?** These patterns are designed for enterprise applications processing thousands of documents. ## Custom Processing Workflows ### Multi-Stage Processing Pipeline ```python Document Classification Pipeline theme={null} from cerevox import Lexa, ProcessingMode import asyncio async def classify_and_process_documents(files): """Classify documents first, then process with appropriate settings""" async with Lexa() as client: # Stage 1: Fast classification pass print("🔍 Stage 1: Document classification...") classification_docs = await client.parse( files, mode=ProcessingMode.DEFAULT # Fast pass for classification ) # Classify documents by type financial_docs = [] legal_docs = [] research_docs = [] for i, doc in enumerate(classification_docs): content_sample = doc.content[:500].lower() if any(word in content_sample for word in ['invoice', 'payment', 'financial', 'amount']): financial_docs.append(files[i]) elif any(word in content_sample for word in ['contract', 'agreement', 'legal', 'party']): legal_docs.append(files[i]) else: research_docs.append(files[i]) print(f"📊 Classified: {len(financial_docs)} financial, {len(legal_docs)} legal, {len(research_docs)} research") # Stage 2: Process each type with optimized settings all_processed = [] if financial_docs: print("💰 Stage 2a: Processing financial documents...") financial_processed = await client.parse( financial_docs, mode=ProcessingMode.ADVANCED, # More accurate but slower for financial data preserve_tables=True, extract_entities=['amounts', 'dates', 'companies'] ) all_processed.extend(financial_processed) if legal_docs: print("⚖️ Stage 2b: Processing legal documents...") legal_processed = await client.parse( legal_docs, mode=ProcessingMode.ADVANCED, preserve_structure=True, extract_entities=['parties', 'dates', 'terms'] ) all_processed.extend(legal_processed) if research_docs: print("📚 Stage 2c: Processing research documents...") research_processed = await client.parse( research_docs, mode=ProcessingMode.DEFAULT, preserve_citations=True, extract_entities=['authors', 'publications', 'data'] ) all_processed.extend(research_processed) print(f"✅ Pipeline complete: {len(all_processed)} documents processed") return all_processed # Process with intelligent classification files = ["invoice.pdf", "contract.docx", "research-paper.pdf"] documents = asyncio.run(classify_and_process_documents(files)) ``` ```python Quality Control Pipeline theme={null} from cerevox import Lexa, LexaError import asyncio async def quality_controlled_processing(files): """Process documents with quality validation and reprocessing""" async def validate_document_quality(doc, source_file): """Validate document processing quality""" quality_score = 0 issues = [] # Check content extraction if len(doc.content) < 100: issues.append("Low content extraction") else: quality_score += 25 # Check table detection (for files that should have tables) if source_file.lower().endswith(('.xlsx', '.csv')) and len(doc.tables) == 0: issues.append("Missing expected tables") else: quality_score += 25 # Check formatting preservation if hasattr(doc, 'formatting_score') and doc.formatting_score > 0.8: quality_score += 25 # Check metadata completeness if doc.metadata and len(doc.metadata) > 5: quality_score += 25 return quality_score, issues async with Lexa() as client: processed_docs = [] retry_files = [] # First processing pass print("🔄 First processing pass...") documents = await client.parse(files, mode=ProcessingMode.DEFAULT) # Quality validation for i, doc in enumerate(documents): quality_score, issues = await validate_document_quality(doc, files[i]) if quality_score >= 75: print(f"✅ {files[i]}: Quality score {quality_score}/100") processed_docs.append(doc) else: print(f"⚠️ {files[i]}: Quality score {quality_score}/100, issues: {issues}") retry_files.append(files[i]) # Reprocess low-quality documents with advanced mode if retry_files: print(f"🔄 Reprocessing {len(retry_files)} documents with advanced mode...") retry_documents = await client.parse( retry_files, mode=ProcessingMode.ADVANCED, timeout=300.0 # Longer timeout for advanced processing ) # Re-validate reprocessed documents for i, doc in enumerate(retry_documents): quality_score, issues = await validate_document_quality(doc, retry_files[i]) print(f"🔄 {retry_files[i]}: Retry quality score {quality_score}/100") processed_docs.append(doc) print(f"✅ Quality control complete: {len(processed_docs)} documents processed") return processed_docs # Process with quality validation files = ["complex-report.pdf", "financial-data.xlsx", "scanned-document.pdf"] documents = asyncio.run(quality_controlled_processing(files)) ``` ## Performance Optimization Patterns ### Intelligent Batching ```python Size-Based Intelligent Batching theme={null} import os from cerevox import Lexa import asyncio async def intelligent_batch_processing(files): """Batch files intelligently based on size and type""" def analyze_files(file_list): """Analyze files to create optimal batches""" file_info = [] for file in file_list: if os.path.exists(file): size = os.path.getsize(file) ext = os.path.splitext(file)[1].lower() # Estimate processing complexity complexity = 1 if ext in ['.pdf', '.docx']: complexity = 2 elif ext in ['.pptx', '.xlsx']: complexity = 3 file_info.append({ 'file': file, 'size': size, 'complexity': complexity, 'estimated_time': size / (1024 * 1024) * complexity # MB * complexity }) return file_info def create_optimal_batches(file_info, max_batch_time=60): """Create batches optimized for processing time""" # Sort by estimated processing time sorted_files = sorted(file_info, key=lambda x: x['estimated_time']) batches = [] current_batch = [] current_time = 0 for file_data in sorted_files: if current_time + file_data['estimated_time'] <= max_batch_time: current_batch.append(file_data['file']) current_time += file_data['estimated_time'] else: if current_batch: batches.append(current_batch) current_batch = [file_data['file']] current_time = file_data['estimated_time'] if current_batch: batches.append(current_batch) return batches # Analyze and batch files file_info = analyze_files(files) batches = create_optimal_batches(file_info) print(f"📊 Created {len(batches)} optimized batches from {len(files)} files") async with Lexa() as client: all_documents = [] for i, batch in enumerate(batches, 1): print(f"🔄 Processing batch {i}/{len(batches)}: {len(batch)} files") batch_start = asyncio.get_event_loop().time() documents = await client.parse(batch) batch_time = asyncio.get_event_loop().time() - batch_start print(f"✅ Batch {i} complete in {batch_time:.2f}s") all_documents.extend(documents) print(f"🎉 Intelligent batching complete: {len(all_documents)} documents") return all_documents # Process with intelligent batching mixed_files = [ "small-text.txt", # 1KB "medium-doc.docx", # 500KB "large-pdf.pdf", # 5MB "complex-sheet.xlsx", # 2MB "presentation.pptx" # 10MB ] documents = asyncio.run(intelligent_batch_processing(mixed_files)) ``` ```python Adaptive Concurrency Control theme={null} from cerevox import Lexa, LexaError import asyncio import time class AdaptiveConcurrencyController: def __init__(self, initial_concurrency=5, min_concurrency=1, max_concurrency=20): self.current_concurrency = initial_concurrency self.min_concurrency = min_concurrency self.max_concurrency = max_concurrency self.success_count = 0 self.error_count = 0 self.last_adjustment = time.time() self.adjustment_interval = 30 # Adjust every 30 seconds def should_adjust(self): return time.time() - self.last_adjustment > self.adjustment_interval def adjust_concurrency(self): if not self.should_adjust(): return total_requests = self.success_count + self.error_count if total_requests == 0: return error_rate = self.error_count / total_requests if error_rate > 0.1: # Too many errors, reduce concurrency self.current_concurrency = max( self.min_concurrency, int(self.current_concurrency * 0.8) ) print(f"🔻 Reducing concurrency to {self.current_concurrency} (error rate: {error_rate:.2%})") elif error_rate < 0.02 and self.success_count > 10: # Low errors, increase concurrency self.current_concurrency = min( self.max_concurrency, int(self.current_concurrency * 1.2) ) print(f"🔺 Increasing concurrency to {self.current_concurrency} (error rate: {error_rate:.2%})") # Reset counters self.success_count = 0 self.error_count = 0 self.last_adjustment = time.time() def record_success(self): self.success_count += 1 self.adjust_concurrency() def record_error(self): self.error_count += 1 self.adjust_concurrency() async def adaptive_processing(files): """Process files with adaptive concurrency control""" controller = AdaptiveConcurrencyController() async def process_with_adaptive_concurrency(client, remaining_files): results = [] while remaining_files: # Get current batch size batch_size = min(controller.current_concurrency, len(remaining_files)) current_batch = remaining_files[:batch_size] remaining_files = remaining_files[batch_size:] print(f"🔄 Processing batch of {len(current_batch)} with concurrency {controller.current_concurrency}") # Process current batch tasks = [process_single_file(client, file) for file in current_batch] batch_results = await asyncio.gather(*tasks, return_exceptions=True) # Record results and adjust concurrency for result in batch_results: if isinstance(result, Exception): controller.record_error() else: controller.record_success() results.append(result) return results async def process_single_file(client, file): try: documents = await client.parse([file]) return documents[0] if documents else None except LexaError as e: print(f"❌ Error processing {file}: {e.message}") raise e async with Lexa() as client: documents = await process_with_adaptive_concurrency(client, files.copy()) print(f"✅ Adaptive processing complete:") print(f" Final concurrency: {controller.current_concurrency}") print(f" Processed: {len(documents)} documents") return documents # Process with adaptive concurrency large_file_list = [f"document_{i:03d}.pdf" for i in range(100)] documents = asyncio.run(adaptive_processing(large_file_list)) ``` ## Custom Content Processing ### Specialized Extraction Patterns ```python Financial Data Extraction theme={null} from cerevox import Lexa import re import asyncio async def extract_financial_insights(files): """Extract structured financial data from documents""" def extract_financial_entities(content): """Extract financial entities from document content""" # Currency amounts pattern currency_pattern = r'\$[\d,]+\.?\d*|\$\d+(?:,\d{3})*(?:\.\d{2})?' amounts = re.findall(currency_pattern, content) # Date patterns date_pattern = r'\b\d{1,2}[\/\-]\d{1,2}[\/\-]\d{2,4}\b|\b\d{4}[\/\-]\d{1,2}[\/\-]\d{1,2}\b' dates = re.findall(date_pattern, content) # Company names (capitalized sequences) company_pattern = r'\b[A-Z][a-z]+ [A-Z][a-z]+(?:\s+(?:Inc|Corp|LLC|Ltd|Co)\.?)?' companies = re.findall(company_pattern, content) # Invoice/Account numbers invoice_pattern = r'(?:Invoice|Account|Reference)\s*#?\s*:?\s*([A-Z0-9\-]+)' numbers = re.findall(invoice_pattern, content, re.IGNORECASE) return { 'amounts': amounts, 'dates': dates, 'companies': list(set(companies)), # Remove duplicates 'reference_numbers': numbers } def analyze_financial_tables(tables): """Analyze tables for financial data patterns""" financial_tables = [] for table in tables: table_analysis = { 'table_index': tables.index(table), 'rows': table.rows, 'columns': table.columns, 'likely_financial': False, 'table_type': 'unknown' } # Analyze table content for financial indicators table_text = str(table.content).lower() financial_keywords = ['amount', 'total', 'subtotal', 'tax', 'payment', 'balance', 'invoice'] financial_score = sum(1 for keyword in financial_keywords if keyword in table_text) if financial_score >= 2: table_analysis['likely_financial'] = True # Determine table type if 'invoice' in table_text: table_analysis['table_type'] = 'invoice' elif 'payment' in table_text: table_analysis['table_type'] = 'payment_schedule' elif 'balance' in table_text: table_analysis['table_type'] = 'balance_sheet' else: table_analysis['table_type'] = 'financial_summary' financial_tables.append(table_analysis) return financial_tables async with Lexa() as client: documents = await client.parse(files, mode=ProcessingMode.ADVANCED) financial_insights = [] for i, doc in enumerate(documents): print(f"💰 Analyzing financial document {i+1}: {files[i]}") # Extract entities from text entities = extract_financial_entities(doc.content) # Analyze tables table_analysis = analyze_financial_tables(doc.tables) # Calculate financial document score financial_score = 0 financial_score += len(entities['amounts']) * 2 financial_score += len(entities['reference_numbers']) * 3 financial_score += sum(1 for table in table_analysis if table['likely_financial']) * 5 insight = { 'source_file': files[i], 'financial_score': financial_score, 'entities': entities, 'table_analysis': table_analysis, 'total_amounts': len(entities['amounts']), 'total_financial_tables': sum(1 for table in table_analysis if table['likely_financial']), 'document_type': 'high_financial' if financial_score > 10 else 'low_financial' } financial_insights.append(insight) print(f" 📊 Financial score: {financial_score}") print(f" 💵 Found {len(entities['amounts'])} amounts") print(f" 📋 Found {len([t for t in table_analysis if t['likely_financial']])} financial tables") return financial_insights # Extract financial insights financial_files = ["invoice.pdf", "financial-statement.xlsx", "payment-report.pdf"] insights = asyncio.run(extract_financial_insights(financial_files)) # Print summary total_amounts = sum(len(insight['entities']['amounts']) for insight in insights) print(f"\n💰 Financial Analysis Summary:") print(f"📄 Documents processed: {len(insights)}") print(f"💵 Total amounts found: {total_amounts}") print(f"📊 High financial documents: {len([i for i in insights if i['document_type'] == 'high_financial'])}") ``` ```python Research Paper Analysis theme={null} from cerevox import Lexa import re import asyncio async def analyze_research_papers(files): """Analyze research papers for academic content""" def extract_academic_entities(content): """Extract academic entities from research content""" # Author patterns author_pattern = r'([A-Z][a-z]+(?:\s+[A-Z]\.)?(?:\s+[A-Z][a-z]+)+)(?:\s+et\s+al\.?)?' authors = re.findall(author_pattern, content) # Citation patterns citation_pattern = r'\[(\d+(?:,\s*\d+)*)\]|$([^)]+\d{4}[^)]*)$' citations = re.findall(citation_pattern, content) # DOI patterns doi_pattern = r'10\.\d{4,}/[^\s]+' dois = re.findall(doi_pattern, content) # Keywords (section headers) keyword_pattern = r'\b(?:Abstract|Introduction|Methodology|Results|Discussion|Conclusion|References)\b' sections = re.findall(keyword_pattern, content, re.IGNORECASE) # Research methods method_keywords = ['experiment', 'survey', 'analysis', 'study', 'research', 'investigation', 'evaluation'] methods = [method for method in method_keywords if method in content.lower()] return { 'authors': list(set(authors)), 'citations': [c for citation_tuple in citations for c in citation_tuple if c], 'dois': dois, 'sections': list(set(sections)), 'methods': methods } def analyze_research_tables(tables): """Analyze tables in research context""" research_tables = [] for table in tables: table_analysis = { 'table_index': tables.index(table), 'rows': table.rows, 'columns': table.columns, 'table_type': 'unknown', 'research_relevance': 0 } table_text = str(table.content).lower() # Research table indicators research_indicators = { 'data': ['mean', 'std', 'deviation', 'correlation', 'p-value', 'significant'], 'results': ['result', 'outcome', 'finding', 'performance', 'accuracy'], 'comparison': ['control', 'experimental', 'baseline', 'comparison', 'vs', 'versus'], 'statistics': ['sample', 'population', 'statistics', 'distribution', 'variance'] } relevance_score = 0 table_types = [] for category, keywords in research_indicators.items(): category_score = sum(1 for keyword in keywords if keyword in table_text) relevance_score += category_score if category_score > 0: table_types.append(category) table_analysis['research_relevance'] = relevance_score table_analysis['table_type'] = ', '.join(table_types) if table_types else 'descriptive' research_tables.append(table_analysis) return research_tables async with Lexa() as client: documents = await client.parse(files, mode=ProcessingMode.ADVANCED) research_analyses = [] for i, doc in enumerate(documents): print(f"📚 Analyzing research paper {i+1}: {files[i]}") # Extract academic entities entities = extract_academic_entities(doc.content) # Analyze tables table_analysis = analyze_research_tables(doc.tables) # Calculate academic quality score academic_score = 0 academic_score += len(entities['authors']) * 2 academic_score += len(entities['citations']) * 1 academic_score += len(entities['dois']) * 5 academic_score += len(entities['sections']) * 3 academic_score += sum(table['research_relevance'] for table in table_analysis) # Determine paper type content_lower = doc.content.lower() paper_type = 'unknown' if 'experiment' in content_lower and 'result' in content_lower: paper_type = 'experimental' elif 'survey' in content_lower or 'review' in content_lower: paper_type = 'survey/review' elif 'theoretical' in content_lower or 'model' in content_lower: paper_type = 'theoretical' elif 'case study' in content_lower: paper_type = 'case_study' analysis = { 'source_file': files[i], 'academic_score': academic_score, 'paper_type': paper_type, 'entities': entities, 'table_analysis': table_analysis, 'total_authors': len(entities['authors']), 'total_citations': len(entities['citations']), 'research_tables': len([t for t in table_analysis if t['research_relevance'] > 2]), 'quality_indicator': 'high' if academic_score > 20 else 'medium' if academic_score > 10 else 'low' } research_analyses.append(analysis) print(f" 📊 Academic score: {academic_score}") print(f" 👥 Found {len(entities['authors'])} authors") print(f" 📖 Found {len(entities['citations'])} citations") print(f" 📋 Found {len([t for t in table_analysis if t['research_relevance'] > 2])} research tables") return research_analyses # Analyze research papers research_files = ["ai-paper.pdf", "machine-learning-study.pdf", "data-science-review.pdf"] analyses = asyncio.run(analyze_research_papers(research_files)) # Print research summary total_citations = sum(len(analysis['entities']['citations']) for analysis in analyses) print(f"\n📚 Research Analysis Summary:") print(f"📄 Papers analyzed: {len(analyses)}") print(f"📖 Total citations: {total_citations}") print(f"🔬 Experimental papers: {len([a for a in analyses if a['paper_type'] == 'experimental'])}") print(f"⭐ High quality papers: {len([a for a in analyses if a['quality_indicator'] == 'high'])}") ``` ## Enterprise Integration Patterns ### Workflow Orchestration ```python Document Processing Workflow theme={null} from cerevox import Lexa import asyncio from datetime import datetime import json class DocumentWorkflow: def __init__(self): self.workflow_id = f"workflow_{datetime.now().strftime('%Y%m%d_%H%M%S')}" self.stages = [] self.results = {} async def execute_stage(self, stage_name, stage_func, *args, **kwargs): """Execute a workflow stage with error handling and logging""" stage_start = datetime.now() print(f"🔄 [{self.workflow_id}] Starting stage: {stage_name}") try: result = await stage_func(*args, **kwargs) stage_end = datetime.now() duration = (stage_end - stage_start).total_seconds() stage_info = { 'name': stage_name, 'status': 'success', 'start_time': stage_start.isoformat(), 'end_time': stage_end.isoformat(), 'duration_seconds': duration, 'result_summary': self._summarize_result(result) } self.stages.append(stage_info) self.results[stage_name] = result print(f"✅ [{self.workflow_id}] Completed stage: {stage_name} ({duration:.2f}s)") return result except Exception as e: stage_end = datetime.now() duration = (stage_end - stage_start).total_seconds() stage_info = { 'name': stage_name, 'status': 'error', 'start_time': stage_start.isoformat(), 'end_time': stage_end.isoformat(), 'duration_seconds': duration, 'error': str(e) } self.stages.append(stage_info) print(f"❌ [{self.workflow_id}] Failed stage: {stage_name} - {e}") raise e def _summarize_result(self, result): """Summarize stage results for logging""" if isinstance(result, list): return f"{len(result)} items" elif hasattr(result, '__len__'): return f"Length: {len(result)}" else: return str(type(result).__name__) def get_workflow_summary(self): """Get complete workflow summary""" total_duration = sum(stage['duration_seconds'] for stage in self.stages) successful_stages = [s for s in self.stages if s['status'] == 'success'] return { 'workflow_id': self.workflow_id, 'total_stages': len(self.stages), 'successful_stages': len(successful_stages), 'total_duration_seconds': total_duration, 'stages': self.stages } async def enterprise_document_workflow(files): """Complete enterprise document processing workflow""" workflow = DocumentWorkflow() # Stage 1: Document Ingestion async def ingestion_stage(files): print(f"📥 Ingesting {len(files)} files...") # Validate files exist and are accessible validated_files = [] for file in files: if os.path.exists(file): validated_files.append(file) else: print(f"⚠️ File not found: {file}") return validated_files # Stage 2: Document Parsing async def parsing_stage(files): async with Lexa() as client: documents = await client.parse(files, mode=ProcessingMode.ADVANCED) return documents # Stage 3: Content Analysis async def analysis_stage(documents): analyzed_docs = [] for doc in documents: analysis = { 'content_length': len(doc.content), 'table_count': len(doc.tables), 'image_count': len(doc.images), 'word_count': len(doc.content.split()), 'complexity_score': len(doc.tables) * 2 + len(doc.images) * 1.5, 'content_type': 'complex' if len(doc.tables) > 5 else 'simple' } analyzed_docs.append({**doc.__dict__, 'analysis': analysis}) return analyzed_docs # Stage 4: Data Transformation async def transformation_stage(analyzed_docs): transformed_data = [] for doc_data in analyzed_docs: # Transform for downstream systems transformed = { 'document_id': f"doc_{len(transformed_data) + 1}", 'source_file': doc_data.get('source_file', 'unknown'), 'processed_content': doc_data['content'][:1000], # First 1000 chars 'metadata': { 'analysis': doc_data['analysis'], 'processing_timestamp': datetime.now().isoformat(), 'workflow_id': workflow.workflow_id }, 'chunks': doc_data.get('text_chunks', [])[:10] # First 10 chunks } transformed_data.append(transformed) return transformed_data # Stage 5: Data Export async def export_stage(transformed_data): export_file = f"workflow_results_{workflow.workflow_id}.json" with open(export_file, 'w') as f: json.dump(transformed_data, f, indent=2, default=str) return { 'export_file': export_file, 'exported_documents': len(transformed_data), 'total_size_mb': os.path.getsize(export_file) / (1024 * 1024) } try: # Execute workflow stages validated_files = await workflow.execute_stage( "ingestion", ingestion_stage, files ) documents = await workflow.execute_stage( "parsing", parsing_stage, validated_files ) analyzed_docs = await workflow.execute_stage( "analysis", analysis_stage, documents ) transformed_data = await workflow.execute_stage( "transformation", transformation_stage, analyzed_docs ) export_result = await workflow.execute_stage( "export", export_stage, transformed_data ) # Generate workflow summary summary = workflow.get_workflow_summary() print(f"\n🎉 Workflow Complete: {workflow.workflow_id}") print(f"📊 Total duration: {summary['total_duration_seconds']:.2f} seconds") print(f"✅ Successful stages: {summary['successful_stages']}/{summary['total_stages']}") print(f"📄 Exported: {export_result['exported_documents']} documents") print(f"💾 Export file: {export_result['export_file']}") return summary, export_result except Exception as e: print(f"💥 Workflow failed: {e}") return workflow.get_workflow_summary(), None # Execute enterprise workflow import os enterprise_files = ["financial-report.pdf", "contracts.docx", "data-analysis.xlsx"] summary, export_result = asyncio.run(enterprise_document_workflow(enterprise_files)) ``` *** **Enterprise Ready:** These patterns are designed for production environments processing thousands of documents. Use them as templates for your own complex workflows. # Async Processing Source: https://docs.cerevox.ai/examples/async-operations Process multiple documents concurrently - 10x faster than sync **Why Async?** Process 100 documents in the time it takes to process 10 synchronously. Essential for high-volume applications. ## Getting Started with Async ### Your First Async Parse ```python Single Document - Async theme={null} import asyncio from cerevox import AsyncLexa async def main(): async with AsyncLexa() as client: # Uses CEREVOX_API_KEY documents = await client.parse("document.pdf") print(f"✅ Async parsing complete: {len(documents[0].content)} chars") # Run it asyncio.run(main()) ``` ```python Multiple Documents - Concurrent theme={null} import asyncio from cerevox import AsyncLexa async def main(): async with AsyncLexa() as client: # Process multiple files concurrently - much faster! documents = await client.parse([ "contract.pdf", "invoice.xlsx", "report.docx" ]) print(f"✅ Processed {len(documents)} documents concurrently") for i, doc in enumerate(documents, 1): print(f" 📄 Doc {i}: {len(doc.content)} chars, {len(doc.tables)} tables") asyncio.run(main()) ``` ```python Batch Processing - Production Ready theme={null} import asyncio from cerevox import AsyncLexa async def process_document_batch(files): async with AsyncLexa() as client: # Process large batches efficiently documents = await client.parse( files, timeout=300.0, # 5 minute timeout for large batches poll_interval=5.0 # Check status every 5 seconds ) return documents # Process 50+ documents efficiently files = [f"documents/doc_{i}.pdf" for i in range(50)] documents = asyncio.run(process_document_batch(files)) print(f"✅ Processed {len(documents)} documents in one batch") ``` ## Real-World Performance Examples ### High-Volume Document Processing ```python Financial Document Processing theme={null} import asyncio from cerevox import AsyncLexa, ProcessingMode async def process_financial_documents(): """Process hundreds of financial documents efficiently""" # Financial documents that need processing financial_docs = [ "invoices/batch_1/*.pdf", # 100+ invoices "statements/q1_2024/*.pdf", # Bank statements "contracts/2024/*.docx", # Legal contracts "reports/financial/*.xlsx" # Financial reports ] async with AsyncLexa() as client: # Process all document types concurrently start_time = asyncio.get_event_loop().time() documents = await client.parse( financial_docs, mode=ProcessingMode.ADVANCED, # More accurate but slower for financial data timeout=600.0 # 10 minute timeout for large batches ) processing_time = asyncio.get_event_loop().time() - start_time print(f"✅ Processed {len(documents)} financial documents") print(f"⚡ Processing time: {processing_time:.2f} seconds") print(f"📊 Average: {processing_time/len(documents):.2f} seconds per document") # Extract structured financial data total_tables = sum(len(doc.tables) for doc in documents) print(f"💰 Extracted {total_tables} financial tables") return documents # Process financial documents at scale documents = asyncio.run(process_financial_documents()) ``` ```python Research Paper Analysis theme={null} import asyncio from cerevox import AsyncLexa async def analyze_research_papers(): """Process academic papers for research analysis""" papers = [ "papers/ai_research_2024/*.pdf", "papers/machine_learning/*.pdf", "papers/data_science/*.pdf" ] async with AsyncLexa() as client: # Process academic papers concurrently documents = await client.parse(papers) # Get research-ready chunks all_chunks = [] for doc in documents: chunks = doc.get_text_chunks( target_size=800, # Larger chunks for research overlap_size=100, # More overlap for context preserve_citations=True # Keep academic citations ) all_chunks.extend(chunks) print(f"📚 Processed {len(documents)} research papers") print(f"🔍 Generated {len(all_chunks)} research chunks") print(f"📊 Found {sum(len(doc.tables) for doc in documents)} data tables") return documents, all_chunks # Analyze research at scale documents, chunks = asyncio.run(analyze_research_papers()) ``` ### RAG System Document Processing ```python Knowledge Base Processing theme={null} import asyncio from cerevox import AsyncLexa async def build_knowledge_base(): """Process documents for RAG knowledge base""" knowledge_docs = [ "knowledge_base/product_docs/*.pdf", "knowledge_base/user_manuals/*.docx", "knowledge_base/faqs/*.html", "knowledge_base/support_articles/*.md" ] async with AsyncLexa() as client: # Process all knowledge base documents documents = await client.parse(knowledge_docs) # Generate RAG-optimized chunks rag_chunks = [] for doc in documents: chunks = doc.get_text_chunks( target_size=500, # Perfect for embeddings overlap_size=50, # Prevent context loss include_metadata=True # Rich metadata for retrieval ) rag_chunks.extend(chunks) print(f"📚 Processed knowledge base: {len(documents)} documents") print(f"🔗 Generated {len(rag_chunks)} RAG chunks") print(f"💾 Ready for vector database: {sum(len(chunk.content) for chunk in rag_chunks)} total characters") # Each chunk is ready for your vector database return rag_chunks # Build your RAG knowledge base rag_chunks = asyncio.run(build_knowledge_base()) # Ready for vector database insertion print(f"✅ {len(rag_chunks)} chunks ready for embedding and storage") ``` ```python Multi-Source RAG Processing theme={null} import asyncio from cerevox import AsyncLexa async def process_multi_source_rag(): """Process documents from multiple sources for comprehensive RAG""" async def process_source(client, source_name, files): """Process documents from a specific source""" documents = await client.parse(files) # Tag chunks with source information chunks = [] for doc in documents: doc_chunks = doc.get_text_chunks(target_size=500) for chunk in doc_chunks: chunk.metadata['source_system'] = source_name chunks.append(chunk) return chunks async with AsyncLexa() as client: # Process multiple sources concurrently tasks = [ process_source(client, "documentation", ["docs/*.pdf"]), process_source(client, "support", ["support/*.docx"]), process_source(client, "knowledge", ["kb/*.html"]), process_source(client, "training", ["training/*.pdf"]) ] # Wait for all sources to complete source_results = await asyncio.gather(*tasks) # Combine all chunks all_chunks = [] for chunks in source_results: all_chunks.extend(chunks) print(f"🔗 Multi-source RAG ready: {len(all_chunks)} chunks") print(f"📊 Sources processed: {len(source_results)}") return all_chunks # Build comprehensive RAG system rag_chunks = asyncio.run(process_multi_source_rag()) ``` ## Controlled Concurrency Patterns ### Production-Grade Concurrency Control ```python Controlled Concurrency theme={null} import asyncio from cerevox import AsyncLexa, LexaError async def process_with_concurrency_limit(files, max_concurrent=5): """Process files with controlled concurrency - prevents overwhelming the API""" semaphore = asyncio.Semaphore(max_concurrent) results = [] async def process_single_file(client, file): async with semaphore: # Limit concurrent operations try: documents = await client.parse([file]) print(f"✅ Processed: {file}") return documents[0] if documents else None except LexaError as e: print(f"❌ Failed {file}: {e.message}") return None async with AsyncLexa() as client: # Create tasks for all files tasks = [process_single_file(client, file) for file in files] # Process with controlled concurrency completed_results = await asyncio.gather(*tasks, return_exceptions=True) # Filter successful results successful_docs = [r for r in completed_results if r and not isinstance(r, Exception)] print(f"✅ Successfully processed {len(successful_docs)}/{len(files)} files") return successful_docs # Process large document sets safely files = [f"documents/batch_{i}.pdf" for i in range(100)] documents = asyncio.run(process_with_concurrency_limit(files, max_concurrent=10)) ``` ```python Batch Processing with Error Recovery theme={null} import asyncio from cerevox import AsyncLexa, LexaError async def robust_batch_processing(all_files, batch_size=20): """Process files in batches with error recovery""" # Split into batches batches = [all_files[i:i + batch_size] for i in range(0, len(all_files), batch_size)] async def process_batch_with_retry(client, batch, batch_num, max_retries=3): for attempt in range(max_retries): try: print(f"🔄 Processing batch {batch_num} (attempt {attempt + 1})") documents = await client.parse(batch, timeout=300.0) print(f"✅ Batch {batch_num} complete: {len(documents)} documents") return documents except LexaError as e: if attempt < max_retries - 1: wait_time = 2 ** attempt # Exponential backoff print(f"⏳ Batch {batch_num} failed, retrying in {wait_time}s...") await asyncio.sleep(wait_time) else: print(f"❌ Batch {batch_num} failed after {max_retries} attempts") return [] return [] async with AsyncLexa() as client: # Process batches with limited concurrency semaphore = asyncio.Semaphore(3) # Max 3 concurrent batches async def controlled_batch_processing(batch, batch_num): async with semaphore: return await process_batch_with_retry(client, batch, batch_num) # Create tasks for all batches tasks = [ controlled_batch_processing(batch, i + 1) for i, batch in enumerate(batches) ] # Process all batches batch_results = await asyncio.gather(*tasks) # Combine successful results all_documents = [] for batch_docs in batch_results: all_documents.extend(batch_docs) print(f"🎉 Batch processing complete: {len(all_documents)} total documents") return all_documents # Process thousands of documents reliably large_file_list = [f"archive/document_{i}.pdf" for i in range(1000)] documents = asyncio.run(robust_batch_processing(large_file_list)) ``` ## Advanced Async Patterns ### Progress Monitoring & Real-time Updates ```python Real-time Progress Tracking theme={null} import asyncio from cerevox import AsyncLexa async def process_with_realtime_progress(files): """Process files with real-time progress updates""" progress_data = { 'total': len(files), 'completed': 0, 'failed': 0, 'in_progress': 0 } def update_progress(status, file_name): """Update progress based on status""" if status == 'started': progress_data['in_progress'] += 1 elif status == 'completed': progress_data['in_progress'] -= 1 progress_data['completed'] += 1 elif status == 'failed': progress_data['in_progress'] -= 1 progress_data['failed'] += 1 # Print progress bar total = progress_data['total'] completed = progress_data['completed'] failed = progress_data['failed'] in_progress = progress_data['in_progress'] progress_pct = (completed + failed) / total * 100 print(f"\r📊 Progress: {progress_pct:.1f}% | ✅ {completed} | ❌ {failed} | 🔄 {in_progress}", end='') async def process_single_with_progress(client, file): update_progress('started', file) try: documents = await client.parse([file]) update_progress('completed', file) return documents[0] if documents else None except Exception as e: update_progress('failed', file) return None async with AsyncLexa() as client: # Process all files with progress tracking tasks = [process_single_with_progress(client, file) for file in files] results = await asyncio.gather(*tasks, return_exceptions=True) print() # New line after progress bar successful_docs = [r for r in results if r and not isinstance(r, Exception)] print(f"🎉 Processing complete!") print(f"✅ Successful: {len(successful_docs)}") print(f"❌ Failed: {len(files) - len(successful_docs)}") return successful_docs # Process with real-time progress files = [f"documents/file_{i}.pdf" for i in range(50)] documents = asyncio.run(process_with_realtime_progress(files)) ``` ```python Queue-based Processing theme={null} import asyncio from asyncio import Queue from cerevox import AsyncLexa async def queue_based_processing(files, num_workers=5): """Process files using a queue with multiple workers""" # Create queues file_queue = Queue() result_queue = Queue() # Add all files to the queue for file in files: await file_queue.put(file) async def worker(client, worker_id): """Worker function to process files from queue""" processed = 0 while True: try: # Get file from queue (timeout after 1 second) file = await asyncio.wait_for(file_queue.get(), timeout=1.0) # Process the file try: documents = await client.parse([file]) await result_queue.put(('success', file, documents)) processed += 1 print(f"👤 Worker {worker_id}: processed {file} ({processed} total)") except Exception as e: await result_queue.put(('error', file, str(e))) print(f"👤 Worker {worker_id}: failed {file}") # Mark task as done file_queue.task_done() except asyncio.TimeoutError: # No more files in queue, worker can exit print(f"👤 Worker {worker_id}: completed ({processed} files processed)") break async with AsyncLexa() as client: # Start worker tasks workers = [ asyncio.create_task(worker(client, i + 1)) for i in range(num_workers) ] # Wait for all files to be processed await file_queue.join() # Cancel workers for w in workers: w.cancel() # Collect results results = [] while not result_queue.empty(): results.append(await result_queue.get()) # Process results successful_docs = [] failed_files = [] for status, file, data in results: if status == 'success': successful_docs.extend(data) else: failed_files.append((file, data)) print(f"🎯 Queue processing complete:") print(f"✅ Successful: {len(successful_docs)}") print(f"❌ Failed: {len(failed_files)}") return successful_docs, failed_files # Process with worker queue files = [f"documents/doc_{i}.pdf" for i in range(100)] successful_docs, failed_files = asyncio.run(queue_based_processing(files, num_workers=8)) ``` ## Integration with Web Frameworks ### FastAPI Integration ```python FastAPI Async Endpoint theme={null} from fastapi import FastAPI, UploadFile, File, BackgroundTasks from cerevox import AsyncLexa import asyncio app = FastAPI() # Global client (reuse connection) lexa_client = None @app.on_event("startup") async def startup_event(): global lexa_client lexa_client = AsyncLexa() @app.on_event("shutdown") async def shutdown_event(): if lexa_client: await lexa_client.close() @app.post("/parse-documents/") async def parse_documents(files: list[UploadFile] = File(...)): """Parse uploaded documents asynchronously""" # Read file contents file_contents = [] for file in files: content = await file.read() file_contents.append(content) # Parse documents concurrently documents = await lexa_client.parse(file_contents) # Return structured results results = [] for i, doc in enumerate(documents): results.append({ 'filename': files[i].filename, 'content_length': len(doc.content), 'tables': len(doc.tables), 'images': len(doc.images), 'content_preview': doc.content[:200] }) return { 'status': 'success', 'processed': len(results), 'results': results } # Run with: uvicorn main:app --reload ``` ```python Background Processing theme={null} from fastapi import FastAPI, BackgroundTasks from cerevox import AsyncLexa import asyncio from typing import Dict app = FastAPI() processing_status: Dict[str, dict] = {} async def background_parse_task(task_id: str, files: list): """Background task for processing large document batches""" processing_status[task_id] = { 'status': 'processing', 'progress': 0, 'total': len(files), 'results': [] } try: async with AsyncLexa() as client: documents = await client.parse(files) # Store results results = [] for doc in documents: results.append({ 'content_length': len(doc.content), 'tables': len(doc.tables), 'chunks': len(doc.get_text_chunks()) }) processing_status[task_id] = { 'status': 'completed', 'progress': 100, 'total': len(files), 'results': results } except Exception as e: processing_status[task_id] = { 'status': 'error', 'error': str(e), 'progress': 0, 'total': len(files) } @app.post("/parse-batch/") async def parse_batch(background_tasks: BackgroundTasks, files: list[str]): """Start background parsing task""" task_id = f"task_{len(processing_status) + 1}" # Start background processing background_tasks.add_task(background_parse_task, task_id, files) return {'task_id': task_id, 'status': 'started'} @app.get("/parse-status/{task_id}") async def get_parse_status(task_id: str): """Get status of parsing task""" if task_id not in processing_status: return {'error': 'Task not found'} return processing_status[task_id] ``` *** **Performance Tip:** Async processing is **10x faster** for multiple documents. Always use async in production for document batches larger than 5 files. # Code Examples Source: https://docs.cerevox.ai/examples/basic-usage Copy-paste ready examples with real data - not toy examples ## Quick Start Examples ### Parse Your First Document ```python Single File - Real Output theme={null} from cerevox import Lexa # Initialize the client client = Lexa() # Uses CEREVOX_API_KEY from environment # Parse a financial document documents = client.parse("invoice.pdf") # Real output - not just {...} doc = documents[0] print(f"✅ Extracted {len(doc.content)} characters") print(f"📊 Found {len(doc.tables)} tables") print(f"💰 Content preview: {doc.content[:200]}...") # Returns actual structured data: # "Invoice #INV-2024-001 # Bill To: Acme Corporation # Amount Due: $1,299.99 # Due Date: 2024-02-15" ``` ```python Multiple Files - Batch Processing theme={null} from cerevox import Lexa client = Lexa() # Process multiple documents efficiently files = [ "contracts/service-agreement.pdf", "invoices/january-2024.xlsx", "reports/quarterly-analysis.docx" ] documents = client.parse(files) # See what you got print(f"✅ Processed {len(documents)} documents") for i, doc in enumerate(documents, 1): print(f" 📄 Document {i}: {len(doc.content)} chars, {len(doc.tables)} tables") # Real results ready for your application ``` ```python Test Content - Perfect for Development theme={null} from cerevox import Lexa client = Lexa() # Parse raw content (great for testing) test_content = b""" INVOICE #12345 Date: 2024-01-15 Bill To: Tech Startup Inc. Amount: $2,499.99 Description: AI Consulting Services """ documents = client.parse(test_content) print(f"✅ Test successful: {documents[0].content}") # Returns exactly what you put in, structured and ready ``` ## Real-World Use Cases ### Financial Document Processing ```python Invoice Extraction theme={null} from cerevox import Lexa client = Lexa() # Parse financial documents documents = client.parse([ "invoices/q1-2024-invoices.pdf", "statements/bank-statement.pdf", "receipts/expense-receipts.xlsx" ]) # Extract key financial data for doc in documents: # Lexa preserves financial formatting print(f"Document: {doc.title}") print(f"Tables found: {len(doc.tables)}") # Tables contain actual structured data if doc.tables: table = doc.tables[0] # First table print(f"Financial data: {table.rows} rows x {table.columns} columns") # Each table has real data, not placeholders # Ready for accounting software integration ``` ```python Contract Analysis theme={null} from cerevox import Lexa client = Lexa() # Parse legal contracts documents = client.parse("contracts/service-agreement.pdf") doc = documents[0] print(f"Contract length: {len(doc.content)} characters") print(f"Key sections preserved: {len(doc.sections)} sections") # Lexa maintains document structure for legal analysis # Perfect for contract review workflows ``` ### Research & Analysis ```python Academic Papers theme={null} from cerevox import Lexa client = Lexa() # Parse research documents documents = client.parse([ "papers/ai-research-2024.pdf", "reports/market-analysis.docx", "data/survey-results.xlsx" ]) # Get structured research data for doc in documents: print(f"📚 Paper: {doc.title}") print(f"📖 Content: {len(doc.content)} chars") print(f"📊 Data tables: {len(doc.tables)} tables") print(f"🖼️ Figures: {len(doc.images)} images") # All formatting and structure preserved ``` ```python Market Reports theme={null} from cerevox import Lexa client = Lexa() # Parse market intelligence documents documents = client.parse("reports/industry-report-2024.pdf") doc = documents[0] print(f"Report sections: {len(doc.sections)}") print(f"Market data tables: {len(doc.tables)}") # Extract market insights with structure intact # Ready for business intelligence tools ``` ## Vector Database Integration ### RAG Application Ready ```python Optimized for Embeddings theme={null} from cerevox import Lexa client = Lexa() # Parse documents for RAG documents = client.parse([ "knowledge-base/product-docs.pdf", "support/troubleshooting.docx" ]) # Get perfectly sized chunks chunks = documents.get_all_text_chunks( target_size=500, # Perfect for most embedding models overlap_size=50, # Prevents context loss include_metadata=True # Rich metadata included ) print(f"🔗 Ready for embedding: {len(chunks)} chunks") # Each chunk is optimized for vector databases for chunk in chunks[:2]: # Show first 2 print(f"\nChunk preview: {chunk.content[:100]}...") print(f"Metadata: page={chunk.page_number}, source={chunk.source_file}") # Rich metadata for better retrieval ``` ```python Direct Vector DB Integration theme={null} from cerevox import Lexa import pinecone # or your preferred vector DB client = Lexa() # Parse and chunk in one step documents = client.parse("knowledge-base/") chunks = documents.get_all_text_chunks(target_size=500) # Ready for your vector database vectors = [] for chunk in chunks: # Each chunk has everything you need vectors.append({ 'id': chunk.id, 'values': your_embedding_model(chunk.content), # Your embedding 'metadata': { 'text': chunk.content, 'page': chunk.page_number, 'source': chunk.source_file, # Rich metadata preserved } }) # Upload to vector database # pinecone.upsert(vectors=vectors) print(f"✅ {len(vectors)} vectors ready for database") ``` ```python Semantic Search Ready theme={null} from cerevox import Lexa client = Lexa() # Parse FAQ documents documents = client.parse("support/faq-database.pdf") # Get semantic search ready chunks chunks = documents.get_all_text_chunks( target_size=300, # Shorter for FAQ preserve_questions=True # Keep Q&A structure ) print(f"🔍 Search ready: {len(chunks)} FAQ chunks") # Each chunk maintains question-answer structure # Perfect for semantic search applications ``` ## Different Input Methods ### File Processing ```python Local Files theme={null} from cerevox import Lexa from pathlib import Path client = Lexa() # Single file doc = client.parse("reports/annual-report.pdf")[0] print(f"✅ Parsed: {len(doc.content)} characters") # Multiple files with Path objects docs_folder = Path("documents") pdf_files = list(docs_folder.glob("*.pdf")) documents = client.parse(pdf_files) print(f"✅ Processed {len(documents)} PDF files") # Mixed file types - Lexa handles them all mixed_files = [ "data.xlsx", # Excel spreadsheet "report.docx", # Word document "slides.pptx", # PowerPoint "data.csv", # CSV file "webpage.html" # HTML file ] documents = client.parse(mixed_files) print(f"✅ Processed {len(documents)} mixed format files") ``` ```python Raw Content Processing theme={null} from cerevox import Lexa from io import BytesIO client = Lexa() # Process bytes directly with open("document.pdf", "rb") as f: content = f.read() documents = client.parse(content) print("✅ Processed raw bytes") # Process file-like objects stream = BytesIO(content) documents = client.parse(stream) print("✅ Processed from stream") # Perfect for web uploads and API integrations ``` ### URL Processing ```python Single URL theme={null} from cerevox import Lexa client = Lexa() # Parse from web URLs url = "https://www.sec.gov/Archives/edgar/data/320193/000032019323000077/aapl-20230930.htm" documents = client.parse_urls(url) print(f"✅ Downloaded and parsed SEC filing") print(f"📄 Content: {len(documents[0].content)} characters") print(f"📊 Tables: {len(documents[0].tables)} financial tables") # Real SEC data, properly structured ``` ```python Multiple URLs - Concurrent Processing theme={null} from cerevox import Lexa client = Lexa() # Process multiple URLs efficiently urls = [ "https://example.com/quarterly-report-q1.pdf", "https://example.com/quarterly-report-q2.pdf", "https://example.com/quarterly-report-q3.pdf" ] documents = client.parse_urls(urls) print(f"✅ Processed {len(documents)} quarterly reports") # All downloaded and parsed concurrently for i, doc in enumerate(documents): print(f" Q{i+1} Report: {len(doc.content)} chars, {len(doc.tables)} tables") ``` ## Processing Modes & Options ### Performance Optimization ```python Processing Modes theme={null} from cerevox import Lexa, ProcessingMode client = Lexa() # Default mode - fast and efficient (recommended) documents = client.parse( "standard-document.pdf", mode=ProcessingMode.DEFAULT # Fast processing for most use cases ) print("✅ Fast processing complete") # Advanced mode - maximum accuracy documents = client.parse( "complex-report.pdf", mode=ProcessingMode.ADVANCED # Use for complex documents requiring maximum accuracy ) print("✅ Advanced processing complete") ``` ```python Progress Tracking theme={null} from cerevox import Lexa def progress_callback(status): print(f"📊 Status: {status.status}") if hasattr(status, 'progress'): print(f"📈 Progress: {status.progress}%") client = Lexa() # Track progress for large jobs documents = client.parse( ["large-file1.pdf", "large-file2.pdf"], progress_callback=progress_callback, timeout=300.0, # 5 minute timeout poll_interval=5.0 # Check every 5 seconds ) print("✅ Large batch processing complete") ``` ### Error Handling ```python Robust Error Handling theme={null} from cerevox import Lexa, LexaError client = Lexa() def safe_parse(files): try: documents = client.parse(files) print(f"✅ Successfully parsed {len(documents)} documents") return documents except LexaError as e: print(f"❌ Lexa API error: {e.message}") if "authentication" in e.message.lower(): print("💡 Check your API key") elif "timeout" in e.message.lower(): print("💡 Try smaller batches or increase timeout") return None except Exception as e: print(f"❌ Unexpected error: {e}") return None # Use with any files documents = safe_parse(["document1.pdf", "document2.docx"]) ``` ```python Retry Logic for Production theme={null} from cerevox import Lexa, LexaError import time client = Lexa() def parse_with_retry(files, max_retries=3): for attempt in range(max_retries): try: documents = client.parse(files) print(f"✅ Success on attempt {attempt + 1}") return documents except LexaError as e: if attempt < max_retries - 1: wait_time = 2 ** attempt # Exponential backoff print(f"⏳ Attempt {attempt + 1} failed, retrying in {wait_time}s...") time.sleep(wait_time) else: print(f"❌ Failed after {max_retries} attempts: {e.message}") raise except Exception as e: print(f"❌ Unexpected error: {e}") raise # Production-ready parsing documents = parse_with_retry(["critical-document.pdf"]) ``` *** **Ready for more?** Check out [async processing](/examples/async-operations) for handling multiple documents concurrently, or [cloud integrations](/examples/cloud-integrations) for S3, SharePoint, and more. # Cloud Integrations Source: https://docs.cerevox.ai/examples/cloud-integrations Parse documents directly from cloud storage services with Lexa ## Supported Cloud Services Parse documents from S3 buckets with IAM integration Access SharePoint sites and document libraries Parse files from Box folders and enterprise content Process documents from Dropbox folders ## Amazon S3 Integration ### Basic S3 Operations ```python List S3 Buckets theme={null} from cerevox import Lexa client = Lexa(api_key="your-api-key") # List all available S3 buckets buckets = client.list_s3_buckets() print(f"Found {len(buckets.buckets)} buckets:") for bucket in buckets.buckets: print(f" 📦 {bucket.name} (Created: {bucket.creation_date})") ``` ```python List S3 Folder Contents theme={null} from cerevox import Lexa client = Lexa(api_key="your-api-key") # List contents of a specific folder contents = client.list_s3_folder( bucket="my-document-bucket", folder_path="invoices/2024/", max_items=100 ) print(f"Found {len(contents.files)} files:") for file in contents.files: print(f" 📄 {file.key} ({file.size} bytes)") print(f" Modified: {file.last_modified}") ``` ```python Parse S3 Documents theme={null} from cerevox import Lexa, ProcessingMode client = Lexa(api_key="your-api-key") # Parse all documents in an S3 folder documents = client.parse_s3_folder( bucket="my-document-bucket", folder_path="contracts/", mode=ProcessingMode.DEFAULT, timeout=300.0 ) print(f"Parsed {len(documents)} documents from S3") for doc in documents: print(f" 📄 {doc.source_file}: {len(doc.content)} characters") ``` ### Advanced S3 Patterns ```python S3 with Progress Monitoring theme={null} from cerevox import Lexa def s3_progress_callback(status): print(f"📊 S3 Processing: {status.status}") if hasattr(status, 'progress') and status.progress: print(f" Progress: {status.progress}%") if hasattr(status, 'files_processed'): processed = getattr(status, 'files_processed', 0) total = getattr(status, 'total_files', 0) print(f" Files: {processed}/{total}") client = Lexa(api_key="your-api-key") # Parse with detailed progress monitoring documents = client.parse_s3_folder( bucket="large-document-bucket", folder_path="annual-reports/", progress_callback=s3_progress_callback, timeout=600.0, # 10 minutes for large batch poll_interval=5.0 ) print(f"✅ Completed: {len(documents)} documents") ``` ```python Selective S3 Processing theme={null} from cerevox import Lexa import fnmatch def process_s3_selectively(bucket, folder_path, file_patterns=None): """Process only specific file types from S3""" client = Lexa(api_key="your-api-key") # List all files first contents = client.list_s3_folder(bucket, folder_path) # Filter files by patterns if file_patterns: filtered_files = [] for file in contents.files: for pattern in file_patterns: if fnmatch.fnmatch(file.key.lower(), pattern): filtered_files.append(file.key) break print(f"Filtered to {len(filtered_files)} files matching patterns: {file_patterns}") else: filtered_files = [f.key for f in contents.files] if not filtered_files: print("No files to process") return [] # Process filtered files in batches batch_size = 20 all_documents = [] for i in range(0, len(filtered_files), batch_size): batch_files = filtered_files[i:i + batch_size] print(f"Processing batch {i//batch_size + 1}: {len(batch_files)} files") try: # Create temporary folder structure for batch batch_folder = f"{folder_path}/batch_{i//batch_size + 1}/" # Note: This is conceptual - actual implementation would # depend on how you want to handle file selection documents = client.parse_s3_folder( bucket=bucket, folder_path=folder_path, timeout=300.0 ) all_documents.extend(documents) except Exception as e: print(f"Batch failed: {e}") continue return all_documents # Usage patterns = ["*.pdf", "*.docx", "*report*"] documents = process_s3_selectively( bucket="document-archive", folder_path="quarterly-reports/", file_patterns=patterns ) ``` ## Microsoft SharePoint Integration ### SharePoint Operations ```python List SharePoint Sites theme={null} from cerevox import Lexa client = Lexa(api_key="your-api-key") # List all available SharePoint sites sites = client.list_sharepoint_sites() print(f"Found {len(sites.sites)} SharePoint sites:") for site in sites.sites: print(f" 🏢 {site.name}") print(f" ID: {site.id}") print(f" URL: {site.web_url}") ``` ```python List SharePoint Drives theme={null} from cerevox import Lexa client = Lexa(api_key="your-api-key") # Get drives for a specific site site_id = "your-site-id" drives = client.list_sharepoint_drives(site_id) print(f"Found {len(drives.drives)} drives:") for drive in drives.drives: print(f" 💾 {drive.name} ({drive.drive_type})") print(f" ID: {drive.id}") print(f" Owner: {drive.owner}") ``` ```python Parse SharePoint Documents theme={null} from cerevox import Lexa, ProcessingMode client = Lexa(api_key="your-api-key") # Parse documents from SharePoint documents = client.parse_sharepoint_folder( site_id="your-site-id", drive_id="your-drive-id", folder_path="Shared Documents/Projects/", mode=ProcessingMode.ADVANCED, timeout=400.0 ) print(f"Processed {len(documents)} SharePoint documents") for doc in documents: print(f" 📄 {doc.title}") print(f" Tables: {len(doc.tables)}") print(f" Images: {len(doc.images)}") ``` ### SharePoint Automation ```python SharePoint Workflow theme={null} from cerevox import Lexa, ProcessingMode import json from datetime import datetime def process_sharepoint_site(site_id, output_dir="sharepoint_results"): """Complete SharePoint site processing workflow""" client = Lexa(api_key="your-api-key") # Step 1: Get all drives in the site print("🔍 Discovering SharePoint structure...") drives = client.list_sharepoint_drives(site_id) all_results = [] for drive in drives.drives: print(f"\n📁 Processing drive: {drive.name}") try: # Parse all documents in the drive documents = client.parse_sharepoint_folder( site_id=site_id, drive_id=drive.id, folder_path="", # Root folder mode=ProcessingMode.DEFAULT, timeout=600.0 ) # Process each document for doc in documents: result = { 'drive_name': drive.name, 'drive_id': drive.id, 'document_title': doc.title, 'source_file': doc.source_file, 'content_length': len(doc.content), 'page_count': doc.page_count, 'tables_count': len(doc.tables), 'images_count': len(doc.images), 'processed_at': datetime.now().isoformat(), 'preview': doc.content[:300] if doc.content else "" } # Extract table summaries if doc.tables: result['table_summary'] = [ { 'rows': table.rows, 'columns': table.columns, 'page': table.page_number } for table in doc.tables ] # Get text chunks for analysis chunks = doc.get_text_chunks(target_size=400) result['chunks_count'] = len(chunks) all_results.append(result) print(f"✅ Processed {len(documents)} documents from {drive.name}") except Exception as e: print(f"❌ Failed to process drive {drive.name}: {e}") continue # Save results output_file = f"{output_dir}/sharepoint_analysis_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json" with open(output_file, 'w') as f: json.dump(all_results, f, indent=2) print(f"\n📊 Analysis complete:") print(f" Total documents: {len(all_results)}") print(f" Results saved to: {output_file}") return all_results # Usage results = process_sharepoint_site("your-sharepoint-site-id") ``` ## Box Integration ### Box Operations ```python List Box Folders theme={null} from cerevox import Lexa client = Lexa(api_key="your-api-key") # List root folders folders = client.list_box_folders() print(f"Root folders in Box:") for folder in folders.folders: print(f" 📁 {folder.name} (ID: {folder.id})") print(f" Created: {folder.created_at}") print(f" Modified: {folder.modified_at}") # List specific folder contents subfolder_id = "123456789" subfolders = client.list_box_folders(subfolder_id) print(f"\nSubfolders in {subfolder_id}:") for folder in subfolders.folders: print(f" 📁 {folder.name}") ``` ```python Parse Box Documents theme={null} from cerevox import Lexa, ProcessingMode client = Lexa(api_key="your-api-key") # Parse all documents in a Box folder folder_id = "your-box-folder-id" documents = client.parse_box_folder( folder_id=folder_id, mode=ProcessingMode.DEFAULT, timeout=300.0 ) print(f"Parsed {len(documents)} documents from Box folder") for doc in documents: print(f" 📄 {doc.source_file}") print(f" Content: {len(doc.content)} characters") if doc.tables: print(f" Tables: {len(doc.tables)}") ``` ## Dropbox Integration ### Dropbox Operations ```python List Dropbox Folders theme={null} from cerevox import Lexa client = Lexa(api_key="your-api-key") # List root folders folders = client.list_dropbox_folders() print("Root folders in Dropbox:") for folder in folders.folders: print(f" 📁 {folder.name}") print(f" Path: {folder.path_display}") # List specific folder specific_folders = client.list_dropbox_folders("/Documents/Work") print(f"\nContents of /Documents/Work:") for folder in specific_folders.folders: print(f" 📁 {folder.name}") ``` ```python Parse Dropbox Documents theme={null} from cerevox import Lexa, ProcessingMode client = Lexa(api_key="your-api-key") # Parse documents from Dropbox folder documents = client.parse_dropbox_folder( folder_path="/Documents/Reports", mode=ProcessingMode.DEFAULT, timeout=300.0 ) print(f"Parsed {len(documents)} documents from Dropbox") for doc in documents: print(f" 📄 {doc.source_file}") print(f" Pages: {doc.page_count}") ``` ## Multi-Cloud Processing ### Unified Cloud Processing ```python Multi-Cloud Processor theme={null} from cerevox import Lexa, ProcessingMode import asyncio from datetime import datetime class MultiCloudProcessor: def __init__(self, api_key): self.client = Lexa(api_key=api_key) self.results = [] def process_s3_source(self, bucket, folder_path=""): """Process documents from S3""" print(f"🔄 Processing S3: s3://{bucket}/{folder_path}") try: documents = self.client.parse_s3_folder( bucket=bucket, folder_path=folder_path, mode=ProcessingMode.DEFAULT, timeout=300.0 ) for doc in documents: self.results.append({ 'source': 'S3', 'location': f"s3://{bucket}/{doc.source_file}", 'document': doc, 'processed_at': datetime.now().isoformat() }) print(f"✅ S3: Processed {len(documents)} documents") return len(documents) except Exception as e: print(f"❌ S3 processing failed: {e}") return 0 def process_sharepoint_source(self, site_id, drive_id, folder_path=""): """Process documents from SharePoint""" print(f"🔄 Processing SharePoint: {site_id}/{drive_id}/{folder_path}") try: documents = self.client.parse_sharepoint_folder( site_id=site_id, drive_id=drive_id, folder_path=folder_path, mode=ProcessingMode.DEFAULT, timeout=300.0 ) for doc in documents: self.results.append({ 'source': 'SharePoint', 'location': f"sharepoint://{site_id}/{drive_id}/{doc.source_file}", 'document': doc, 'processed_at': datetime.now().isoformat() }) print(f"✅ SharePoint: Processed {len(documents)} documents") return len(documents) except Exception as e: print(f"❌ SharePoint processing failed: {e}") return 0 def process_box_source(self, folder_id): """Process documents from Box""" print(f"🔄 Processing Box: {folder_id}") try: documents = self.client.parse_box_folder( folder_id=folder_id, mode=ProcessingMode.DEFAULT, timeout=300.0 ) for doc in documents: self.results.append({ 'source': 'Box', 'location': f"box://{folder_id}/{doc.source_file}", 'document': doc, 'processed_at': datetime.now().isoformat() }) print(f"✅ Box: Processed {len(documents)} documents") return len(documents) except Exception as e: print(f"❌ Box processing failed: {e}") return 0 def process_dropbox_source(self, folder_path): """Process documents from Dropbox""" print(f"🔄 Processing Dropbox: {folder_path}") try: documents = self.client.parse_dropbox_folder( folder_path=folder_path, mode=ProcessingMode.DEFAULT, timeout=300.0 ) for doc in documents: self.results.append({ 'source': 'Dropbox', 'location': f"dropbox://{folder_path}/{doc.source_file}", 'document': doc, 'processed_at': datetime.now().isoformat() }) print(f"✅ Dropbox: Processed {len(documents)} documents") return len(documents) except Exception as e: print(f"❌ Dropbox processing failed: {e}") return 0 def get_summary(self): """Get processing summary""" by_source = {} total_docs = len(self.results) total_content = 0 total_tables = 0 for result in self.results: source = result['source'] doc = result['document'] if source not in by_source: by_source[source] = { 'count': 0, 'content_chars': 0, 'tables': 0 } by_source[source]['count'] += 1 by_source[source]['content_chars'] += len(doc.content) by_source[source]['tables'] += len(doc.tables) total_content += len(doc.content) total_tables += len(doc.tables) return { 'total_documents': total_docs, 'total_content_chars': total_content, 'total_tables': total_tables, 'by_source': by_source } # Usage processor = MultiCloudProcessor(api_key="your-api-key") # Process from multiple cloud sources processor.process_s3_source("my-s3-bucket", "documents/") processor.process_sharepoint_source("site-id", "drive-id", "Shared Documents/") processor.process_box_source("box-folder-id") processor.process_dropbox_source("/Work Documents") # Get summary summary = processor.get_summary() print(f"\n📊 Multi-Cloud Processing Summary:") print(f" Total documents: {summary['total_documents']}") print(f" Total content: {summary['total_content_chars']} characters") print(f" Total tables: {summary['total_tables']}") for source, stats in summary['by_source'].items(): print(f"\n {source}:") print(f" Documents: {stats['count']}") print(f" Content: {stats['content_chars']} chars") print(f" Tables: {stats['tables']}") ``` ## Production Cloud Patterns ### Robust Cloud Processing ```python Production Cloud Pipeline theme={null} from cerevox import Lexa, LexaError, ProcessingMode import logging import time from pathlib import Path import json logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) class ProductionCloudProcessor: def __init__(self, api_key, config=None): self.client = Lexa( api_key=api_key, timeout=300.0, max_retries=3 ) self.config = config or { 'max_retries': 3, 'retry_delay': 2.0, 'batch_size': 20, 'processing_mode': ProcessingMode.DEFAULT } self.stats = { 'total_processed': 0, 'total_failed': 0, 'total_retries': 0, 'processing_time': 0 } def process_with_resilience(self, process_func, *args, **kwargs): """Execute cloud processing with retry logic""" max_retries = self.config['max_retries'] retry_delay = self.config['retry_delay'] for attempt in range(max_retries): try: start_time = time.time() result = process_func(*args, **kwargs) processing_time = time.time() - start_time self.stats['processing_time'] += processing_time self.stats['total_processed'] += len(result) if result else 0 logger.info(f"✅ Processing successful: {len(result) if result else 0} documents") return result except LexaError as e: logger.warning(f"Attempt {attempt + 1} failed: {e.message}") if attempt < max_retries - 1: if e.error_code == "RATE_LIMIT_EXCEEDED": wait_time = int(getattr(e, 'retry_after', retry_delay * (2 ** attempt))) else: wait_time = retry_delay * (2 ** attempt) logger.info(f"Retrying in {wait_time} seconds...") time.sleep(wait_time) self.stats['total_retries'] += 1 else: logger.error(f"❌ All attempts failed: {e.message}") self.stats['total_failed'] += 1 raise e except Exception as e: logger.error(f"❌ Unexpected error: {e}") self.stats['total_failed'] += 1 if attempt == max_retries - 1: raise e time.sleep(retry_delay * (2 ** attempt)) self.stats['total_retries'] += 1 return None def process_s3_resilient(self, bucket, folder_path=""): """Resilient S3 processing""" return self.process_with_resilience( self.client.parse_s3_folder, bucket=bucket, folder_path=folder_path, mode=self.config['processing_mode'], timeout=300.0 ) def process_sharepoint_resilient(self, site_id, drive_id, folder_path=""): """Resilient SharePoint processing""" return self.process_with_resilience( self.client.parse_sharepoint_folder, site_id=site_id, drive_id=drive_id, folder_path=folder_path, mode=self.config['processing_mode'], timeout=300.0 ) def save_checkpoint(self, results, checkpoint_file): """Save processing checkpoint""" checkpoint_data = { 'results': [ { 'source_file': doc.source_file, 'content_length': len(doc.content), 'tables_count': len(doc.tables), 'images_count': len(doc.images) } for doc in results ], 'stats': self.stats, 'timestamp': time.time() } with open(checkpoint_file, 'w') as f: json.dump(checkpoint_data, f, indent=2) logger.info(f"💾 Checkpoint saved: {checkpoint_file}") def get_processing_report(self): """Generate processing report""" return { 'summary': { 'total_processed': self.stats['total_processed'], 'total_failed': self.stats['total_failed'], 'total_retries': self.stats['total_retries'], 'success_rate': ( self.stats['total_processed'] / (self.stats['total_processed'] + self.stats['total_failed']) if (self.stats['total_processed'] + self.stats['total_failed']) > 0 else 0 ) * 100, 'total_processing_time': self.stats['processing_time'], 'avg_processing_time': ( self.stats['processing_time'] / self.stats['total_processed'] if self.stats['total_processed'] > 0 else 0 ) }, 'recommendations': self._get_recommendations() } def _get_recommendations(self): """Get performance recommendations""" recommendations = [] success_rate = ( self.stats['total_processed'] / (self.stats['total_processed'] + self.stats['total_failed']) if (self.stats['total_processed'] + self.stats['total_failed']) > 0 else 0 ) * 100 if success_rate < 90: recommendations.append("Consider increasing retry limits or timeout values") if self.stats['total_retries'] > self.stats['total_processed'] * 0.5: recommendations.append("High retry rate detected - check network connectivity") avg_time = ( self.stats['processing_time'] / self.stats['total_processed'] if self.stats['total_processed'] > 0 else 0 ) if avg_time > 10: recommendations.append("Consider using FAST processing mode for better performance") return recommendations # Usage processor = ProductionCloudProcessor( api_key="your-api-key", config={ 'max_retries': 5, 'retry_delay': 3.0, 'processing_mode': ProcessingMode.DEFAULT } ) # Process with resilience try: s3_docs = processor.process_s3_resilient("my-bucket", "documents/") processor.save_checkpoint(s3_docs, "s3_checkpoint.json") sharepoint_docs = processor.process_sharepoint_resilient( "site-id", "drive-id", "Shared Documents/" ) processor.save_checkpoint(sharepoint_docs, "sharepoint_checkpoint.json") except Exception as e: logger.error(f"Critical failure: {e}") # Generate report report = processor.get_processing_report() print(f"\n📊 Processing Report:") print(f" Success Rate: {report['summary']['success_rate']:.1f}%") print(f" Total Processed: {report['summary']['total_processed']}") print(f" Total Failed: {report['summary']['total_failed']}") print(f" Average Time: {report['summary']['avg_processing_time']:.1f}s per document") if report['recommendations']: print(f"\n💡 Recommendations:") for rec in report['recommendations']: print(f" • {rec}") ``` *** Explore [advanced patterns](/examples/advanced-patterns) for sophisticated document processing workflows. # Best Practices Source: https://docs.cerevox.ai/guides/best-practices Production-ready patterns and best practices for Lexa ## 🚀 Quick Wins Use AsyncLexa for 3-5x better performance in production Process multiple documents together for optimal throughput Leverage built-in chunking for vector database optimization Implement proper retry logic and graceful degradation ## Production Patterns ### Async-First Architecture Always prefer async operations for production workloads: ```python Recommended: Async theme={null} import asyncio from cerevox import AsyncLexa, ProcessingMode async def process_documents(file_paths): async with AsyncLexa(api_key="your-api-key") as client: # Process multiple documents concurrently documents = await client.parse( file_paths, mode=ProcessingMode.DEFAULT ) # Get vector-ready chunks chunks = documents.get_all_text_chunks(target_size=512) return chunks # Process 100+ documents efficiently file_batch = ["doc1.pdf", "doc2.docx", "doc3.txt"] chunks = asyncio.run(process_documents(file_batch)) ``` ```python Avoid: Sync for Large Batches theme={null} from cerevox import Lexa # This blocks the thread for each document client = Lexa(api_key="your-api-key") for file_path in file_paths: # Sequential processing document = client.parse(file_path) # Blocking call ``` ### Robust Error Handling Implement comprehensive error handling with automatic retries: ```python theme={null} import asyncio from cerevox import AsyncLexa, LexaError from tenacity import retry, stop_after_attempt, wait_exponential class ProductionLexaClient: def __init__(self, api_key: str): self.api_key = api_key @retry( stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10) ) async def safe_parse(self, files, **kwargs): """Parse with automatic retry on failures""" try: async with AsyncLexa(api_key=self.api_key) as client: return await client.parse(files, **kwargs) except LexaError as e: if e.status_code >= 500: # Server errors - retry raise else: # Client errors - don't retry return self._handle_client_error(e) def _handle_client_error(self, error): """Handle client errors gracefully""" print(f"Client error: {error.message}") return [] # Return empty result or default # Usage client = ProductionLexaClient("your-api-key") documents = await client.safe_parse(["document.pdf"]) ``` ### Progress Monitoring Implement detailed progress tracking for long-running operations: ```python theme={null} from datetime import datetime import logging def production_progress_callback(status): """Production-ready progress callback with logging""" timestamp = datetime.now().isoformat() # Log to your monitoring system logging.info(f"[{timestamp}] Job {status.job_id}: {status.status}") if hasattr(status, 'progress') and status.progress: logging.info(f"Progress: {status.progress}") # Send to monitoring service (DataDog, New Relic, etc.) # monitor.track('lexa.processing.progress', status.progress) async def monitored_parsing(): async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse( ["large_document.pdf"], progress_callback=production_progress_callback, timeout=300.0, # 5 minutes poll_interval=2.0 # Check every 2 seconds ) return documents ``` ## Performance Optimization ### Optimal Batch Sizing Balance throughput and memory usage: ```python theme={null} async def optimize_batch_processing(file_paths: list): """Process files in optimal batch sizes""" OPTIMAL_BATCH_SIZE = 10 # Adjust based on file sizes async with AsyncLexa(api_key="your-api-key") as client: all_documents = [] for i in range(0, len(file_paths), OPTIMAL_BATCH_SIZE): batch = file_paths[i:i + OPTIMAL_BATCH_SIZE] # Process batch documents = await client.parse( batch, mode=ProcessingMode.DEFAULT # Fast processing for most use cases ) all_documents.extend(documents) # Optional: Brief pause to avoid rate limits await asyncio.sleep(0.1) return all_documents ``` ### Processing Mode Selection Choose the right mode for your use case: ```python Fast Processing (Default) theme={null} # Best for: Most documents, general content processing documents = await client.parse( files, mode=ProcessingMode.DEFAULT # Fast and efficient ) ``` ```python Maximum Accuracy (Advanced) theme={null} # Best for: Complex layouts, research papers, critical documents documents = await client.parse( files, mode=ProcessingMode.ADVANCED # Highest accuracy ) ``` ### Memory Management Handle large document batches efficiently: ```python theme={null} async def memory_efficient_processing(file_paths): """Process large batches without memory issues""" async with AsyncLexa(api_key="your-api-key") as client: for batch in chunk_files(file_paths, batch_size=5): # Process batch documents = await client.parse(batch) # Process immediately and release memory chunks = documents.get_all_text_chunks(target_size=512) # Send to vector DB immediately await store_in_vector_db(chunks) # Explicitly release memory del documents, chunks def chunk_files(files, batch_size): """Split files into batches""" for i in range(0, len(files), batch_size): yield files[i:i + batch_size] ``` ## Vector Database Integration ### Optimal Chunking Strategy Configure chunking for your vector database: ```python theme={null} async def prepare_for_vector_db(documents): """Optimize chunks for different vector databases""" # For OpenAI embeddings (ada-002) openai_chunks = documents.get_all_text_chunks( target_size=1500, # ~2000 tokens with overhead tolerance=0.1 # 10% size tolerance ) # For sentence transformers sentence_chunks = documents.get_all_text_chunks( target_size=384, # Typical context window tolerance=0.15 ) # For document-level embeddings doc_chunks = documents.get_all_text_chunks( target_size=2000, # Larger chunks for context tolerance=0.2 ) return { 'openai': openai_chunks, 'sentence': sentence_chunks, 'document': doc_chunks } ``` ### Rich Metadata Extraction Include comprehensive metadata for better retrieval: ```python theme={null} def extract_rich_metadata(documents): """Extract comprehensive metadata for vector storage""" chunks_with_metadata = [] for doc in documents: text_chunks = doc.get_text_chunks(target_size=512) for chunk in text_chunks: metadata = { # Document metadata 'filename': doc.filename, 'file_type': doc.file_type, 'total_pages': doc.total_pages, 'processing_date': datetime.now().isoformat(), # Chunk metadata 'chunk_size': len(chunk), 'chunk_index': text_chunks.index(chunk), 'total_chunks': len(text_chunks), # Content metadata 'has_tables': len(doc.tables) > 0, 'has_images': len(doc.images) > 0, 'element_count': doc.total_elements, # Custom fields 'document_category': classify_document(doc), 'extraction_confidence': calculate_confidence(doc) } chunks_with_metadata.append({ 'content': chunk, 'metadata': metadata }) return chunks_with_metadata ``` ## Security Best Practices ### API Key Management Never hardcode API keys in your application: ```python ✅ Secure: Environment Variables theme={null} import os from cerevox import AsyncLexa # Load from environment api_key = os.getenv('CEREVOX_API_KEY') if not api_key: raise ValueError("CEREVOX_API_KEY environment variable not set") async with AsyncLexa(api_key=api_key) as client: documents = await client.parse(files) ``` ```python ✅ Secure: Configuration Files theme={null} import json from pathlib import Path def load_config(): config_path = Path.home() / '.cerevox' / 'config.json' with open(config_path) as f: return json.load(f) config = load_config() client = AsyncLexa(api_key=config['api_key']) ``` ```python ❌ Insecure: Hardcoded Keys theme={null} # Never do this! client = AsyncLexa(api_key="sk-1234567890abcdef") ``` ### Data Privacy Handle sensitive documents securely: ```python theme={null} import tempfile import os from pathlib import Path async def secure_document_processing(sensitive_files): """Process sensitive documents with automatic cleanup""" temp_dir = None try: # Create secure temporary directory temp_dir = tempfile.mkdtemp(prefix='cerevox_secure_') # Process documents async with AsyncLexa(api_key=os.getenv('CEREVOX_API_KEY')) as client: documents = await client.parse(sensitive_files) # Process results immediately chunks = documents.get_all_text_chunks(target_size=512) # Store in secure location secure_results = encrypt_and_store(chunks) return secure_results finally: # Always cleanup temporary files if temp_dir and Path(temp_dir).exists(): import shutil shutil.rmtree(temp_dir, ignore_errors=True) ``` ## Monitoring and Observability ### Production Logging Implement comprehensive logging: ```python theme={null} import logging import time from functools import wraps # Configure structured logging logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s' ) logger = logging.getLogger('cerevox.production') def monitor_performance(func): """Decorator to monitor Lexa operations""" @wraps(func) async def wrapper(*args, **kwargs): start_time = time.time() operation_name = func.__name__ try: logger.info(f"Starting {operation_name}") result = await func(*args, **kwargs) duration = time.time() - start_time logger.info(f"Completed {operation_name} in {duration:.2f}s") # Send metrics to monitoring service # metrics.timing(f'cerevox.{operation_name}.duration', duration) return result except Exception as e: duration = time.time() - start_time logger.error(f"Failed {operation_name} after {duration:.2f}s: {e}") # Track errors # metrics.increment(f'cerevox.{operation_name}.error') raise return wrapper @monitor_performance async def parse_documents_monitored(files): async with AsyncLexa(api_key=os.getenv('CEREVOX_API_KEY')) as client: return await client.parse(files) ``` ### Health Checks Implement health monitoring: ```python theme={null} async def health_check(): """Health check for Lexa service availability""" try: async with AsyncLexa(api_key=os.getenv('CEREVOX_API_KEY')) as client: # Test with minimal document test_content = b"Health check test document." start_time = time.time() documents = await client.parse(test_content, timeout=10.0) response_time = time.time() - start_time return { 'status': 'healthy', 'response_time': response_time, 'service': 'cerevox-lexa' } except Exception as e: return { 'status': 'unhealthy', 'error': str(e), 'service': 'cerevox-lexa' } # Use in your health endpoint # @app.get("/health/cerevox") # async def cerevox_health(): # return await health_check() ``` ## Testing Strategies ### Unit Testing Test your Lexa integration thoroughly: ```python theme={null} import pytest from unittest.mock import AsyncMock, patch from cerevox import AsyncLexa, LexaError class TestLexaIntegration: @pytest.fixture async def client(self): async with AsyncLexa(api_key="test-key") as client: yield client @pytest.mark.asyncio async def test_successful_parsing(self, client): """Test successful document parsing""" test_file = b"Test document content" with patch.object(client, 'parse') as mock_parse: mock_parse.return_value = [Mock(content="Parsed content")] result = await client.parse(test_file) assert len(result) == 1 assert result[0].content == "Parsed content" @pytest.mark.asyncio async def test_error_handling(self, client): """Test error handling""" with patch.object(client, 'parse') as mock_parse: mock_parse.side_effect = LexaError("API Error", status_code=400) with pytest.raises(LexaError): await client.parse("test.pdf") @pytest.mark.asyncio async def test_chunking_output(self, client): """Test chunking functionality""" # Test with mock document mock_doc = Mock() mock_doc.get_text_chunks.return_value = ["chunk1", "chunk2"] chunks = mock_doc.get_text_chunks(target_size=512) assert len(chunks) == 2 assert chunks[0] == "chunk1" ``` Remember to test with realistic document sizes and types that match your production workload. ## Common Pitfalls ### Avoid These Mistakes **Don't process files sequentially** - Use async operations and batch processing for better performance. **Don't ignore timeouts** - Set appropriate timeouts based on your document sizes and processing requirements. **Don't skip error handling** - Always implement proper error handling and retry logic for production systems. ### Performance Anti-Patterns ```python theme={null} # ❌ Anti-pattern: Sequential processing for file in large_file_list: doc = client.parse(file) # Blocks thread process(doc) # ✅ Better: Batch async processing async with AsyncLexa(api_key=api_key) as client: documents = await client.parse(large_file_list) for doc in documents: process(doc) ``` ## Production Checklist Before deploying to production: * [ ] API keys stored securely (environment variables) * [ ] Timeout values configured appropriately * [ ] Processing modes selected for use case * [ ] Batch sizes optimized for your workload * [ ] Retry logic implemented * [ ] Graceful degradation on failures * [ ] Comprehensive error logging * [ ] Health checks in place * [ ] Async operations used throughout * [ ] Memory management for large batches * [ ] Progress monitoring implemented * [ ] Performance metrics tracked * [ ] No hardcoded credentials * [ ] Secure temporary file handling * [ ] Data privacy measures * [ ] Access controls configured ## Next Steps Learn advanced patterns for RAG applications Deep dive into performance tuning Explore the complete API documentation See advanced implementation patterns # Performance Optimization Source: https://docs.cerevox.ai/guides/performance-optimization Maximize Lexa performance for high-volume document processing **Performance Goal:** Process 1000+ documents efficiently while maintaining accuracy and minimizing costs. ## Quick Performance Wins ### Processing Mode Optimization ```python Choose the Right Mode theme={null} from cerevox import Lexa, ProcessingMode client = Lexa() # ✅ DEFAULT mode - fast and efficient for most documents documents = client.parse( "documents/*.pdf", mode=ProcessingMode.DEFAULT # Fast processing for most use cases ) # ✅ ADVANCED mode - maximum accuracy for complex documents documents = client.parse( "complex-research-papers/*.pdf", mode=ProcessingMode.ADVANCED # Use for complex layouts, research papers ) print("💡 Rule: Start with DEFAULT, use ADVANCED for complex docs requiring maximum accuracy") ``` ```python Mode Performance Comparison theme={null} # Real performance data from Lexa: # DEFAULT Mode: # - Speed: ~3 seconds per document # - Best for: Most documents, general content # - Accuracy: 96% for typical documents # - Resource usage: Efficient # ADVANCED Mode: # - Speed: ~8 seconds per document # - Best for: Complex layouts, research papers, technical docs # - Accuracy: 99%+ for complex documents # - Resource usage: Higher # Choose based on your accuracy vs speed requirements ``` ### Async Processing (10x Faster) ```python Async vs Sync Performance theme={null} import asyncio import time from cerevox import Lexa, AsyncLexa def sync_processing_slow(files): """Slow synchronous processing""" client = Lexa() start_time = time.time() all_documents = [] for file in files: documents = client.parse([file]) # One at a time all_documents.extend(documents) end_time = time.time() print(f"😴 Sync processing: {end_time - start_time:.2f} seconds") return all_documents async def async_processing_fast(files): """Fast asynchronous processing""" async with AsyncLexa() as client: start_time = time.time() # Process all files concurrently documents = await client.parse(files) end_time = time.time() print(f"🚀 Async processing: {end_time - start_time:.2f} seconds") return documents # Performance comparison files = [f"document_{i}.pdf" for i in range(20)] # Sync: ~100 seconds for 20 files sync_docs = sync_processing_slow(files) # Async: ~10 seconds for 20 files (10x faster!) async_docs = asyncio.run(async_processing_fast(files)) print("💡 Always use async for multiple documents!") ``` ```python Optimal Concurrency Settings theme={null} import asyncio from cerevox import AsyncLexa async def find_optimal_concurrency(files): """Find the optimal concurrency for your use case""" concurrency_levels = [1, 5, 10, 15, 20] for concurrency in concurrency_levels: start_time = time.time() async with AsyncLexa() as client: # Process with limited concurrency semaphore = asyncio.Semaphore(concurrency) async def process_with_limit(file): async with semaphore: return await client.parse([file]) tasks = [process_with_limit(file) for file in files[:20]] # Test with 20 files results = await asyncio.gather(*tasks) end_time = time.time() processing_time = end_time - start_time print(f"Concurrency {concurrency:2d}: {processing_time:.2f}s ({20/processing_time:.1f} docs/sec)") # Find your optimal settings test_files = [f"test_doc_{i}.pdf" for i in range(20)] asyncio.run(find_optimal_concurrency(test_files)) # Typical results: # Concurrency 1: 45.2s (0.4 docs/sec) # Concurrency 5: 12.1s (1.7 docs/sec) ← Often optimal # Concurrency 10: 8.5s (2.4 docs/sec) ← Good for larger files # Concurrency 15: 9.2s (2.2 docs/sec) ← Diminishing returns # Concurrency 20: 11.1s (1.8 docs/sec) ← Too high, performance drops ``` ## Batch Processing Strategies ### Intelligent Batching ```python Size-Based Batching theme={null} import os from cerevox import AsyncLexa import asyncio async def intelligent_batching(files): """Batch files based on size for optimal performance""" def analyze_files(file_list): file_info = [] for file in file_list: if os.path.exists(file): size_mb = os.path.getsize(file) / (1024 * 1024) # Categorize by size if size_mb < 1: category = 'small' batch_size = 50 # Small files: large batches elif size_mb < 10: category = 'medium' batch_size = 20 # Medium files: moderate batches else: category = 'large' batch_size = 5 # Large files: small batches file_info.append({ 'file': file, 'size_mb': size_mb, 'category': category, 'batch_size': batch_size }) return file_info # Analyze and group files file_info = analyze_files(files) # Group by category categories = {} for info in file_info: category = info['category'] if category not in categories: categories[category] = [] categories[category].append(info['file']) async with AsyncLexa() as client: all_documents = [] for category, category_files in categories.items(): batch_size = file_info[0]['batch_size'] if file_info else 20 print(f"📋 Processing {len(category_files)} {category} files in batches of {batch_size}") for i in range(0, len(category_files), batch_size): batch = category_files[i:i + batch_size] start_time = time.time() documents = await client.parse(batch) batch_time = time.time() - start_time all_documents.extend(documents) docs_per_sec = len(documents) / batch_time print(f" ✅ {category} batch: {len(documents)} docs in {batch_time:.2f}s ({docs_per_sec:.1f} docs/sec)") return all_documents # Example usage mixed_files = [ "small-invoice.pdf", # 100KB "medium-report.pdf", # 5MB "large-presentation.pdf" # 25MB ] documents = asyncio.run(intelligent_batching(mixed_files)) ``` ```python Memory-Efficient Processing theme={null} import asyncio from cerevox import AsyncLexa import gc async def memory_efficient_processing(large_file_list, chunk_size=100): """Process large datasets without memory issues""" total_processed = 0 # Process in chunks to manage memory for i in range(0, len(large_file_list), chunk_size): chunk = large_file_list[i:i + chunk_size] print(f"🔄 Processing chunk {i//chunk_size + 1}: {len(chunk)} files") async with AsyncLexa() as client: documents = await client.parse(chunk) # Process documents immediately (save to DB, extract data, etc.) processed_data = [] for doc in documents: # Extract only what you need processed_data.append({ 'source_file': doc.source_file, 'content_length': len(doc.content), 'table_count': len(doc.tables), 'summary': doc.content[:500] # Only first 500 chars }) # Save processed data await save_to_database(processed_data) # Your save function total_processed += len(documents) # Clear variables and force garbage collection del documents del processed_data gc.collect() print(f" ✅ Chunk complete. Total processed: {total_processed}") # Brief pause between chunks await asyncio.sleep(0.5) print(f"🎉 Memory-efficient processing complete: {total_processed} documents") async def save_to_database(processed_data): """Placeholder for your database save function""" # Implement your database save logic here await asyncio.sleep(0.1) # Simulate save time # Process 10,000 documents efficiently large_dataset = [f"document_{i:05d}.pdf" for i in range(10000)] asyncio.run(memory_efficient_processing(large_dataset)) ``` ## Error Handling & Retry Strategies ### Production-Ready Error Handling ```python Robust Error Handling theme={null} from cerevox import AsyncLexa, LexaError import asyncio import time async def robust_processing_with_retries(files, max_retries=3): """Production-ready processing with intelligent retries""" async def process_with_retry(client, file, attempt=0): try: documents = await client.parse([file]) return {'file': file, 'documents': documents, 'success': True} except LexaError as e: if attempt < max_retries: # Exponential backoff wait_time = (2 ** attempt) print(f"⏳ Retry {attempt + 1} for {file} in {wait_time}s: {e.message}") await asyncio.sleep(wait_time) return await process_with_retry(client, file, attempt + 1) else: print(f"❌ Max retries exceeded for {file}: {e.message}") return {'file': file, 'error': str(e), 'success': False} except Exception as e: print(f"💥 Unexpected error for {file}: {e}") return {'file': file, 'error': str(e), 'success': False} async with AsyncLexa() as client: # Process all files with retries tasks = [process_with_retry(client, file) for file in files] results = await asyncio.gather(*tasks, return_exceptions=True) # Analyze results successful = [r for r in results if isinstance(r, dict) and r['success']] failed = [r for r in results if isinstance(r, dict) and not r['success']] exceptions = [r for r in results if isinstance(r, Exception)] print(f"📊 Processing Results:") print(f" ✅ Successful: {len(successful)}") print(f" ❌ Failed: {len(failed)}") print(f" 💥 Exceptions: {len(exceptions)}") return successful, failed, exceptions # Process with robust error handling files_with_issues = ["good-doc.pdf", "corrupted-doc.pdf", "missing-doc.pdf"] successful, failed, exceptions = asyncio.run(robust_processing_with_retries(files_with_issues)) ``` ```python Circuit Breaker Pattern theme={null} import asyncio from cerevox import AsyncLexa import time class CircuitBreaker: def __init__(self, failure_threshold=5, recovery_timeout=60): self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.failure_count = 0 self.last_failure_time = None self.state = 'CLOSED' # CLOSED, OPEN, HALF_OPEN async def call(self, coro): if self.state == 'OPEN': if time.time() - self.last_failure_time > self.recovery_timeout: self.state = 'HALF_OPEN' print("🔄 Circuit breaker: HALF_OPEN (trying recovery)") else: raise Exception("Circuit breaker is OPEN") try: result = await coro await self._on_success() return result except Exception as e: await self._on_failure() raise e async def _on_success(self): self.failure_count = 0 if self.state == 'HALF_OPEN': self.state = 'CLOSED' print("✅ Circuit breaker: CLOSED (recovery successful)") async def _on_failure(self): self.failure_count += 1 self.last_failure_time = time.time() if self.failure_count >= self.failure_threshold: self.state = 'OPEN' print(f"🔴 Circuit breaker: OPEN ({self.failure_count} failures)") async def process_with_circuit_breaker(files): """Process files with circuit breaker protection""" circuit_breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=30) async with AsyncLexa() as client: results = [] for file in files: try: documents = await circuit_breaker.call(client.parse([file])) results.append({'file': file, 'documents': documents, 'success': True}) print(f"✅ Processed: {file}") except Exception as e: if "Circuit breaker is OPEN" in str(e): print(f"🔴 Skipped {file}: Circuit breaker open") results.append({'file': file, 'error': 'Circuit breaker open', 'success': False}) else: print(f"❌ Failed: {file} - {e}") results.append({'file': file, 'error': str(e), 'success': False}) return results # Use circuit breaker for unstable processing scenarios test_files = ["doc1.pdf", "doc2.pdf", "problematic-doc.pdf", "doc4.pdf"] results = asyncio.run(process_with_circuit_breaker(test_files)) ``` ## Cost Optimization ### Smart Processing Strategies ```python Cost-Effective Processing theme={null} from cerevox import Lexa, ProcessingMode import time def cost_optimized_processing(files): """Optimize for cost while maintaining quality""" client = Lexa() # Strategy 1: Use DEFAULT mode for simple documents simple_extensions = ['.txt', '.csv', '.md'] complex_extensions = ['.pdf', '.docx', '.pptx'] simple_files = [f for f in files if any(f.endswith(ext) for ext in simple_extensions)] complex_files = [f for f in files if any(f.endswith(ext) for ext in complex_extensions)] all_documents = [] # Process simple files with DEFAULT mode (cheaper) if simple_files: print(f"💰 Processing {len(simple_files)} simple files with DEFAULT mode") start_time = time.time() simple_docs = client.parse( simple_files, mode=ProcessingMode.DEFAULT # Fast processing for most use cases ) processing_time = time.time() - start_time print(f" ✅ DEFAULT mode: {len(simple_docs)} docs in {processing_time:.2f}s") all_documents.extend(simple_docs) # Process complex files with ADVANCED mode (balanced cost/quality) if complex_files: print(f"📄 Processing {len(complex_files)} complex files with ADVANCED mode") start_time = time.time() complex_docs = client.parse( complex_files, mode=ProcessingMode.ADVANCED # Use for complex layouts, research papers ) processing_time = time.time() - start_time print(f" ✅ ADVANCED mode: {len(complex_docs)} docs in {processing_time:.2f}s") all_documents.extend(complex_docs) print(f"💡 Cost optimization: Used DEFAULT mode for {len(simple_files)} files, ADVANCED for {len(complex_files)} files") return all_documents # Cost optimization example mixed_files = [ "simple-data.txt", # Use DEFAULT mode "simple-list.csv", # Use DEFAULT mode "complex-report.pdf", # Use ADVANCED mode "presentation.pptx" # Use ADVANCED mode ] documents = cost_optimized_processing(mixed_files) ``` ```python Batch Size Optimization theme={null} import asyncio from cerevox import AsyncLexa import time async def optimize_batch_sizes(files): """Find optimal batch sizes for cost and performance""" batch_sizes = [5, 10, 20, 50] # Test different batch sizes for batch_size in batch_sizes: print(f"🧪 Testing batch size: {batch_size}") start_time = time.time() total_docs = 0 async with AsyncLexa() as client: for i in range(0, min(len(files), 100), batch_size): # Test with first 100 files batch = files[i:i + batch_size] batch_start = time.time() documents = await client.parse(batch) batch_time = time.time() - batch_start total_docs += len(documents) # Cost calculation (example - adjust based on your pricing) cost_per_doc = 0.01 # $0.01 per document batch_cost = len(documents) * cost_per_doc print(f" Batch {i//batch_size + 1}: {len(documents)} docs, {batch_time:.2f}s, ${batch_cost:.2f}") total_time = time.time() - start_time docs_per_second = total_docs / total_time cost_per_hour = docs_per_second * 3600 * 0.01 # Hourly cost estimate print(f" 📊 Batch size {batch_size}: {docs_per_second:.1f} docs/sec, ~${cost_per_hour:.2f}/hour") print() # Find optimal batch size for your use case large_file_set = [f"doc_{i:03d}.pdf" for i in range(500)] asyncio.run(optimize_batch_sizes(large_file_set)) # Typical results: # Batch size 5: 2.1 docs/sec, ~$75.60/hour # Batch size 10: 3.8 docs/sec, ~$136.80/hour ← Often optimal # Batch size 20: 4.2 docs/sec, ~$151.20/hour # Batch size 50: 3.9 docs/sec, ~$140.40/hour ← Diminishing returns ``` ## Performance Monitoring ### Real-Time Performance Tracking ```python Performance Monitor theme={null} import time import asyncio from cerevox import AsyncLexa class PerformanceMonitor: def __init__(self): self.stats = { 'total_documents': 0, 'total_time': 0, 'successful_documents': 0, 'failed_documents': 0, 'average_doc_size': 0, 'docs_per_second': 0 } self.start_time = None def start_monitoring(self): self.start_time = time.time() print("📊 Performance monitoring started") def record_batch(self, documents, processing_time): self.stats['total_documents'] += len(documents) self.stats['total_time'] += processing_time self.stats['successful_documents'] += len(documents) # Calculate running averages if self.stats['total_time'] > 0: self.stats['docs_per_second'] = self.stats['total_documents'] / self.stats['total_time'] # Calculate average document size total_content = sum(len(doc.content) for doc in documents) self.stats['average_doc_size'] = total_content / len(documents) if documents else 0 def record_failure(self, failed_count): self.stats['failed_documents'] += failed_count def print_stats(self): print(f"\n📊 Performance Statistics:") print(f" 📄 Total documents: {self.stats['total_documents']}") print(f" ✅ Successful: {self.stats['successful_documents']}") print(f" ❌ Failed: {self.stats['failed_documents']}") print(f" ⚡ Speed: {self.stats['docs_per_second']:.2f} docs/second") print(f" 📏 Avg doc size: {self.stats['average_doc_size']:,.0f} chars") print(f" ⏱️ Total time: {self.stats['total_time']:.2f} seconds") if self.start_time: elapsed = time.time() - self.start_time print(f" 🕐 Elapsed time: {elapsed:.2f} seconds") async def monitored_processing(files, batch_size=20): """Process files with performance monitoring""" monitor = PerformanceMonitor() monitor.start_monitoring() async with AsyncLexa() as client: for i in range(0, len(files), batch_size): batch = files[i:i + batch_size] batch_start = time.time() try: documents = await client.parse(batch) batch_time = time.time() - batch_start monitor.record_batch(documents, batch_time) print(f"✅ Batch {i//batch_size + 1}: {len(documents)} docs in {batch_time:.2f}s") except Exception as e: batch_time = time.time() - batch_start monitor.record_failure(len(batch)) print(f"❌ Batch {i//batch_size + 1} failed: {e}") # Print stats every 5 batches if (i // batch_size + 1) % 5 == 0: monitor.print_stats() # Final statistics monitor.print_stats() return monitor.stats # Monitor processing performance test_files = [f"document_{i:03d}.pdf" for i in range(100)] stats = asyncio.run(monitored_processing(test_files)) ``` ## Performance Best Practices * **DEFAULT mode**: Fast processing for most use cases * **ADVANCED mode**: Maximum accuracy for complex documents * **Sweet spot**: 5-10 concurrent requests for most use cases * **Small files**: Can handle 15-20 concurrent requests * **Large files**: Reduce to 3-5 concurrent requests * **Monitor**: Watch for rate limiting and adjust accordingly * **Small files** (\< 1MB): Batches of 30-50 files * **Medium files** (1-10MB): Batches of 10-20 files * **Large files** (>10MB): Batches of 3-5 files * **Mixed sizes**: Group by size before batching * Process large datasets in chunks (100-500 files per chunk) * Clear document variables after processing * Use garbage collection for long-running processes * Save results immediately, don't accumulate in memory *** **Performance Rule:** Start with async processing + DEFAULT mode + batches of 20. This gives 80% of optimal performance with minimal tuning. # Vector Database Integration Source: https://docs.cerevox.ai/guides/vector-database-integration Ready-to-use patterns for popular vector databases - Pinecone, Weaviate, Chroma, and more Optimized 500-char chunks ideal for most embedding models Page numbers, source files, document structure preserved Works out-of-the-box with popular vector databases Battle-tested patterns for enterprise RAG systems ## Quick RAG Setup (5 Minutes) ### Your First RAG Knowledge Base ```python Parse and Chunk - Vector DB Ready theme={null} from cerevox import Lexa client = Lexa() # Parse your knowledge base documents documents = client.parse([ "product-docs/*.pdf", "user-manuals/*.docx", "faqs/*.html" ]) # Get vector DB optimized chunks chunks = documents.get_all_text_chunks( target_size=500, # Perfect for most embeddings overlap_size=50, # Prevents context loss include_metadata=True # Rich metadata included ) print(f"✅ Ready for vector database: {len(chunks)} chunks") # Each chunk has everything you need: for chunk in chunks[:2]: print(f"Text: {chunk.content[:100]}...") print(f"Page: {chunk.page_number}") print(f"Source: {chunk.source_file}") print(f"Metadata: {chunk.metadata}") print("---") ``` ```python Real Output Example theme={null} # What you actually get from Lexa: { 'content': 'Our API supports both REST and GraphQL endpoints. Authentication is handled via API keys that can be generated from your dashboard. Rate limits apply based on your subscription tier.', 'page_number': 5, 'source_file': 'api-documentation.pdf', 'metadata': { 'chunk_id': 'api-docs_chunk_12', 'document_title': 'API Documentation v2.1', 'section': 'Authentication', 'chunk_index': 12, 'total_chunks': 45 } } # Perfect for vector databases - no post-processing needed! ``` ## Vector Database Examples ### Pinecone Integration ```python Complete Pinecone Setup theme={null} import pinecone from cerevox import Lexa from sentence_transformers import SentenceTransformer # 1. Setup Pinecone pinecone.init( api_key="your-pinecone-key", environment="us-west1-gcp" # Your environment ) # Create index index_name = "knowledge-base" if index_name not in pinecone.list_indexes(): pinecone.create_index( name=index_name, dimension=384, # For all-MiniLM-L6-v2 metric="cosine" ) index = pinecone.Index(index_name) # 2. Setup embedding model embedder = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2') # 3. Parse and upload documents client = Lexa() documents = client.parse(["knowledge-base/*.pdf"]) # Get optimized chunks chunks = documents.get_all_text_chunks(target_size=500) # 4. Upload to Pinecone vectors_to_upsert = [] for chunk in chunks: # Create embedding embedding = embedder.encode(chunk.content).tolist() # Prepare for Pinecone vectors_to_upsert.append({ 'id': f"{chunk.source_file}_{chunk.page_number}_{len(vectors_to_upsert)}", 'values': embedding, 'metadata': { 'text': chunk.content, 'source': chunk.source_file, 'page': chunk.page_number, # All Lexa metadata preserved **chunk.metadata } }) # Upload in batches batch_size = 100 for i in range(0, len(vectors_to_upsert), batch_size): batch = vectors_to_upsert[i:i + batch_size] index.upsert(vectors=batch) print(f"✅ Uploaded {len(vectors_to_upsert)} vectors to Pinecone") ``` ```python Pinecone RAG Query theme={null} def query_knowledge_base(question, top_k=5): """Query your Pinecone knowledge base""" # Embed the question question_embedding = embedder.encode(question).tolist() # Search Pinecone results = index.query( vector=question_embedding, top_k=top_k, include_metadata=True ) # Extract relevant chunks relevant_chunks = [] for match in results['matches']: relevant_chunks.append({ 'text': match['metadata']['text'], 'source': match['metadata']['source'], 'page': match['metadata']['page'], 'score': match['score'] }) return relevant_chunks # Example usage question = "How do I authenticate with the API?" results = query_knowledge_base(question) for result in results: print(f"📄 Source: {result['source']} (Page {result['page']})") print(f"📊 Score: {result['score']:.3f}") print(f"💬 Text: {result['text'][:200]}...") print("---") ``` ### Weaviate Integration ```python Weaviate Complete Setup theme={null} import weaviate from cerevox import Lexa # 1. Connect to Weaviate client_weaviate = weaviate.Client( url="https://your-cluster.weaviate.network", auth_client_secret=weaviate.AuthApiKey(api_key="your-key") ) # 2. Create schema schema = { "classes": [{ "class": "KnowledgeChunk", "description": "Document chunks from Lexa parsing", "vectorizer": "text2vec-openai", # or your preferred vectorizer "properties": [ { "name": "content", "dataType": ["text"], "description": "The chunk text content" }, { "name": "source_file", "dataType": ["string"], "description": "Source document filename" }, { "name": "page_number", "dataType": ["int"], "description": "Page number in source document" }, { "name": "chunk_index", "dataType": ["int"], "description": "Index of chunk in document" } ] }] } # Create schema (run once) try: client_weaviate.schema.create(schema) print("✅ Weaviate schema created") except: print("ℹ️ Schema already exists") # 3. Parse and upload documents lexa_client = Lexa() documents = lexa_client.parse(["documents/*.pdf"]) chunks = documents.get_all_text_chunks(target_size=500) # 4. Upload to Weaviate with client_weaviate.batch as batch: batch.batch_size = 100 for chunk in chunks: batch.add_data_object( data_object={ "content": chunk.content, "source_file": chunk.source_file, "page_number": chunk.page_number, "chunk_index": getattr(chunk, 'chunk_index', 0) }, class_name="KnowledgeChunk" ) print(f"✅ Uploaded {len(chunks)} chunks to Weaviate") ``` ```python Weaviate RAG Query theme={null} def query_weaviate(question, limit=5): """Query Weaviate knowledge base""" result = ( client_weaviate.query .get("KnowledgeChunk", ["content", "source_file", "page_number"]) .with_near_text({"concepts": [question]}) .with_additional(["certainty"]) .with_limit(limit) .do() ) chunks = result["data"]["Get"]["KnowledgeChunk"] for chunk in chunks: print(f"📄 Source: {chunk['source_file']} (Page {chunk['page_number']})") print(f"📊 Certainty: {chunk['_additional']['certainty']:.3f}") print(f"💬 Content: {chunk['content'][:200]}...") print("---") return chunks # Query the knowledge base results = query_weaviate("What are the authentication methods?") ``` ### Chroma Integration ```python Chroma Simple Setup theme={null} import chromadb from cerevox import Lexa # 1. Initialize Chroma chroma_client = chromadb.Client() # Create collection collection = chroma_client.create_collection( name="knowledge_base", metadata={"description": "Lexa processed documents"} ) # 2. Parse documents with Lexa lexa_client = Lexa() documents = lexa_client.parse(["docs/*.pdf", "manuals/*.docx"]) chunks = documents.get_all_text_chunks(target_size=500) # 3. Prepare data for Chroma documents_list = [] metadatas_list = [] ids_list = [] for i, chunk in enumerate(chunks): documents_list.append(chunk.content) metadatas_list.append({ "source_file": chunk.source_file, "page_number": chunk.page_number, "chunk_type": "text" }) ids_list.append(f"chunk_{i}") # 4. Add to Chroma collection.add( documents=documents_list, metadatas=metadatas_list, ids=ids_list ) print(f"✅ Added {len(chunks)} chunks to Chroma") ``` ```python Chroma RAG Queries theme={null} def query_chroma(question, n_results=5): """Query Chroma knowledge base""" results = collection.query( query_texts=[question], n_results=n_results, include=["documents", "metadatas", "distances"] ) for i, doc in enumerate(results['documents'][0]): metadata = results['metadatas'][0][i] distance = results['distances'][0][i] print(f"📄 Source: {metadata['source_file']} (Page {metadata['page_number']})") print(f"📊 Distance: {distance:.3f}") print(f"💬 Content: {doc[:200]}...") print("---") return results # Query example results = query_chroma("How do I configure the settings?") ``` ## Production RAG Patterns ### Advanced Chunking Strategies ```python Multi-Modal RAG Setup theme={null} from cerevox import Lexa def create_multimodal_chunks(files): """Create specialized chunks for different content types""" client = Lexa() documents = client.parse(files) all_chunks = [] for doc in documents: # Regular text chunks text_chunks = doc.get_text_chunks(target_size=500) for chunk in text_chunks: all_chunks.append({ 'content': chunk.content, 'type': 'text', 'source': chunk.source_file, 'page': chunk.page_number, 'metadata': chunk.metadata }) # Table-specific chunks (larger for context) for table in doc.tables: table_content = f"Table from page {table.page_number}:\n{table.to_text()}" if table.caption: table_content = f"Table Caption: {table.caption}\n{table_content}" all_chunks.append({ 'content': table_content, 'type': 'table', 'source': doc.source_file, 'page': table.page_number, 'metadata': { 'rows': table.rows, 'columns': table.columns, 'table_id': table.id } }) # Image descriptions (if available) for image in doc.images: if hasattr(image, 'description') and image.description: all_chunks.append({ 'content': f"Image description: {image.description}", 'type': 'image', 'source': doc.source_file, 'page': image.page_number, 'metadata': { 'image_id': image.id, 'alt_text': getattr(image, 'alt_text', '') } }) print(f"📊 Created multimodal chunks:") print(f" 📝 Text: {len([c for c in all_chunks if c['type'] == 'text'])}") print(f" 📋 Tables: {len([c for c in all_chunks if c['type'] == 'table'])}") print(f" 🖼️ Images: {len([c for c in all_chunks if c['type'] == 'image'])}") return all_chunks # Create multimodal knowledge base multimodal_chunks = create_multimodal_chunks(["complex-report.pdf"]) ``` ```python Hierarchical Chunking theme={null} from cerevox import Lexa def create_hierarchical_chunks(files): """Create hierarchical chunks with document structure""" client = Lexa() documents = client.parse(files) hierarchical_chunks = [] for doc in documents: # Document-level chunk (summary) doc_summary = doc.content[:1000] # First 1000 chars as summary hierarchical_chunks.append({ 'content': doc_summary, 'level': 'document', 'source': doc.source_file, 'metadata': { 'total_pages': doc.page_count, 'total_content_length': len(doc.content), 'chunk_type': 'document_summary' } }) # Section-level chunks (if sections detected) if hasattr(doc, 'sections') and doc.sections: for section in doc.sections: hierarchical_chunks.append({ 'content': section.content, 'level': 'section', 'source': doc.source_file, 'metadata': { 'section_title': section.title, 'section_number': section.number, 'parent_document': doc.source_file } }) # Paragraph-level chunks text_chunks = doc.get_text_chunks(target_size=300) # Smaller for paragraphs for i, chunk in enumerate(text_chunks): hierarchical_chunks.append({ 'content': chunk.content, 'level': 'paragraph', 'source': chunk.source_file, 'page': chunk.page_number, 'metadata': { 'chunk_index': i, 'parent_document': doc.source_file, 'paragraph_type': 'content' } }) return hierarchical_chunks # Create hierarchical structure hierarchical_chunks = create_hierarchical_chunks(["structured-document.pdf"]) print(f"📊 Hierarchical chunks created:") for level in ['document', 'section', 'paragraph']: count = len([c for c in hierarchical_chunks if c['level'] == level]) print(f" {level.title()}: {count} chunks") ``` ### High-Performance RAG Pipeline ```python Production RAG Pipeline theme={null} import asyncio from cerevox import AsyncLexa from concurrent.futures import ThreadPoolExecutor import time class ProductionRAGPipeline: def __init__(self, vector_db_client, embedding_model): self.vector_db = vector_db_client self.embedder = embedding_model self.processed_docs = set() async def process_documents_async(self, files, batch_size=20): """Process documents in parallel batches""" async with AsyncLexa() as client: print(f"🚀 Processing {len(files)} documents in batches of {batch_size}") all_chunks = [] # Process in batches for i in range(0, len(files), batch_size): batch = files[i:i + batch_size] print(f"📋 Processing batch {i//batch_size + 1}: {len(batch)} files") start_time = time.time() # Parse documents documents = await client.parse(batch) # Create chunks batch_chunks = [] for doc in documents: chunks = doc.get_text_chunks(target_size=500) batch_chunks.extend(chunks) all_chunks.extend(batch_chunks) batch_time = time.time() - start_time print(f"✅ Batch complete: {len(batch_chunks)} chunks in {batch_time:.2f}s") return all_chunks async def upload_to_vector_db_async(self, chunks, batch_size=100): """Upload chunks to vector database with threading""" def embed_batch(batch_chunks): """Embed a batch of chunks (CPU intensive)""" texts = [chunk.content for chunk in batch_chunks] embeddings = self.embedder.encode(texts) return embeddings print(f"🔗 Creating embeddings for {len(chunks)} chunks...") # Use ThreadPoolExecutor for CPU-intensive embedding with ThreadPoolExecutor(max_workers=4) as executor: upload_futures = [] for i in range(0, len(chunks), batch_size): batch = chunks[i:i + batch_size] # Create embeddings in thread future = executor.submit(embed_batch, batch) upload_futures.append((batch, future)) # Process results and upload for batch, future in upload_futures: embeddings = future.result() # Prepare vectors for upload vectors = [] for chunk, embedding in zip(batch, embeddings): vectors.append({ 'id': f"{chunk.source_file}_{chunk.page_number}_{len(vectors)}", 'values': embedding.tolist(), 'metadata': { 'text': chunk.content, 'source': chunk.source_file, 'page': chunk.page_number } }) # Upload to vector database await self.upload_vectors_async(vectors) print(f"✅ All chunks uploaded to vector database") async def upload_vectors_async(self, vectors): """Upload vectors to database (implement for your vector DB)""" # Implement based on your vector database # This is a placeholder for async upload await asyncio.sleep(0.1) # Simulate upload time print(f"📤 Uploaded batch of {len(vectors)} vectors") # Usage example async def run_production_pipeline(): # Initialize your vector DB and embedding model # vector_db = YourVectorDBClient() # embedder = YourEmbeddingModel() # pipeline = ProductionRAGPipeline(vector_db, embedder) # Large document set large_doc_set = [f"documents/doc_{i:04d}.pdf" for i in range(1000)] # Process documents start_time = time.time() # chunks = await pipeline.process_documents_async(large_doc_set) # await pipeline.upload_to_vector_db_async(chunks) total_time = time.time() - start_time print(f"🎉 Production pipeline complete in {total_time:.2f} seconds") # print(f"📊 Processed {len(chunks)} chunks from {len(large_doc_set)} documents") # Run the production pipeline # asyncio.run(run_production_pipeline()) ``` ## Vector Database Comparison **Best for:** Production applications, auto-scaling, minimal setup ```python theme={null} # Pros: Fully managed, excellent performance, auto-scaling # Cons: Cost scales with usage, vendor lock-in # Use when: Building production RAG applications ``` **Best for:** Flexibility, custom schemas, hybrid search ```python theme={null} # Pros: Open source, hybrid search, flexible schemas # Cons: More complex setup, resource intensive # Use when: Need hybrid search or custom data models ``` **Best for:** Development, small to medium datasets ```python theme={null} # Pros: Simple setup, lightweight, great for development # Cons: Limited scalability for very large datasets # Use when: Prototyping or smaller applications ``` **Best for:** High-performance requirements, filtering ```python theme={null} # Pros: Excellent performance, advanced filtering, Rust-based # Cons: Newer ecosystem, fewer integrations # Use when: Performance is critical ``` *** **RAG Ready:** Lexa chunks work out-of-the-box with any vector database. Start with Chroma for development, then scale to Pinecone or Weaviate for production. # Best Practices Source: https://docs.cerevox.ai/hippo/best-practices Optimize RAG quality, performance, and cost savings # Hippo Best Practices Maximize answer quality while achieving **80% cost reduction** with these proven strategies. ## Document Preparation ### Upload High-Quality Documents **Prefer**: Text-based PDFs (created from Word, Google Docs, etc.) **Avoid**: Scanned/image PDFs (OCR quality varies) ```python theme={null} # Check if PDF is text-based import PyPDF2 def is_text_pdf(file_path): with open(file_path, 'rb') as f: pdf = PyPDF2.PdfReader(f) text = pdf.pages[0].extract_text() return len(text.strip()) > 50 # Has extractable text if is_text_pdf("document.pdf"): hippo.upload_file(folder_id, "document.pdf") else: print("Warning: Scanned PDF - consider OCR first") ``` **Impact**: 30-40% better accuracy with text-based PDFs Before uploading, remove: * Cover pages and blank pages * Table of contents (unless needed for answers) * Advertisements and promotional material * Appendices with irrelevant data **Impact**: Faster processing + less noise in answers **Good formatting**: * Clear headings and structure * Proper paragraph breaks * Readable fonts (not decorative) * Logical document flow **Bad formatting**: * All-caps text * Excessive formatting * Broken layouts * Mixed languages without context **Impact**: Better chunk quality → Better retrieval ## Folder Organization ### Strategic Document Grouping **✅ Good**: All product docs in one folder ```python theme={null} product_folder = hippo.create_folder("Product V2 Docs") hippo.upload_file(product_folder.id, "features.pdf") hippo.upload_file(product_folder.id, "api.pdf") hippo.upload_file(product_folder.id, "examples.pdf") ``` **Impact**: Better cross-document answers **✅ Good**: Separate folders for different products ```python theme={null} product_a_folder = hippo.create_folder("Product A") product_b_folder = hippo.create_folder("Product B") ``` **❌ Bad**: Mix all products in one folder **Impact**: Reduced confusion, better precision ### Folder Size Sweet Spot ```python theme={null} # Optimal folder sizes for best performance folder_guidelines = { "Small": "5-20 documents", # Fast, focused "Medium": "20-100 documents", # Recommended "Large": "100-500 documents", # Still good "Very Large": "500+ documents" # Consider splitting } ``` **Recommendation**: 20-100 related documents per folder for best results ## Question Optimization ### Write Clear, Specific Questions **✅ Good**: * "What is the API rate limit for Pro plan users?" * "What is the refund window for digital products?" * "What authentication methods does the API support?" **❌ Bad**: * "Tell me about limits" * "Refunds?" * "Auth" **Impact**: 2-3x better answer relevance **✅ Good**: * "How do I integrate Stripe payment processing?" * "How can I export user data to CSV format?" * "How do I configure SSO with Okta?" **❌ Bad**: * "Stripe setup" * "Export data" * "SSO" **Impact**: Step-by-step answers vs vague responses **✅ Good**: * "What's the difference between Basic and Pro plans?" * "How does REST API compare to GraphQL API?" * "Which deployment option is recommended for high traffic?" **❌ Bad**: * "Plans?" * "REST vs GraphQL" * "Deployment" **Impact**: Comprehensive comparisons vs incomplete answers ### Leverage Follow-Up Questions ```python theme={null} # Use conversation context for follow-ups def conversational_qa(hippo, chat_id): # Q1: Establish context a1 = hippo.submit_ask( chat_id, "What are the API authentication methods?" ) print(f"Q1: {a1.response}\n") # Q2: Follow-up (uses Q1 context) a2 = hippo.submit_ask( chat_id, "Which one is most secure?" # Refers to "methods" from Q1 ) print(f"Q2: {a2.response}\n") # Q3: Another follow-up (uses Q2 context) a3 = hippo.submit_ask( chat_id, "How do I implement it?" # Refers to "secure method" from Q2 ) print(f"Q3: {a3.response}\n") return [a1, a2, a3] ``` **Impact**: Natural conversation flow → Better understanding ## Performance Optimization ### Use Async for Scale ```python Sync - Sequential (Slower) theme={null} # Sequential uploads - 30 seconds for file in files: hippo.upload_file(folder_id, file) ``` ```python Async - Concurrent (Faster) theme={null} import asyncio from cerevox import AsyncHippo # Concurrent uploads - 5 seconds async with AsyncHippo() as hippo: tasks = [hippo.upload_file(folder_id, f) for f in files] await asyncio.gather(*tasks) ``` **Impact**: 5-10x faster batch operations ### Batch Related Questions ```python theme={null} import asyncio async def batch_qa(hippo, chat_id, questions): """Ask multiple questions concurrently""" tasks = [ hippo.submit_ask(chat_id, q) for q in questions ] answers = await asyncio.gather(*tasks) return answers # Usage questions = [ "What is the API rate limit?", "What are the supported file formats?", "How do I authenticate?" ] async with AsyncHippo() as hippo: answers = await batch_qa(hippo, chat_id, questions) for q, a in zip(questions, answers): print(f"Q: {q}") print(f"A: {a.response}\n") ``` **Impact**: 3-5x faster for multiple independent questions ## Cost Optimization ### Maximize the 80% Savings ```python theme={null} # ✅ Upload documents once folder = hippo.create_folder("Docs") hippo.upload_file(folder.id, "guide.pdf") # ✅ Ask many questions (cost-effective) chat = hippo.create_chat(folder.id) for question in questions: answer = hippo.submit_ask(chat.id, question) ``` **Impact**: Amortize upload cost over many queries ```python theme={null} # ✅ Reuse chat for related questions support_chat = hippo.create_chat(folder.id, "Support") # Multiple user questions use same chat for user_question in user_questions: answer = hippo.submit_ask(support_chat.id, user_question) ``` **Impact**: Maintain context, reduce overhead ### Precision Retrieval Benefits Hippo automatically retrieves only relevant chunks: ```python theme={null} # Traditional RAG full_docs = load_documents() # 50,000 tokens cost_traditional = 50_000 * $0.001 # $0.05 per query # Hippo RAG answer = hippo.submit_ask(chat_id, question) # → Retrieves ~15,000 tokens (70% smaller) cost_hippo = 15_000 * $0.0002 # $0.003 per query # Savings: 80% reduction print(f"Traditional: ${cost_traditional:.3f}") print(f"Hippo: ${cost_hippo:.3f}") print(f"Savings: {(1 - cost_hippo/cost_traditional)*100:.0f}%") ``` ## Answer Quality ### Verify with Confidence Scores ```python theme={null} def get_verified_answer(hippo, chat_id, question): """Get answer with confidence verification""" answer = hippo.submit_ask(chat_id, question) if answer.confidence_score >= 0.9: status = "✅ High confidence" elif answer.confidence_score >= 0.7: status = "⚠️ Medium confidence - verify sources" else: status = "❌ Low confidence - may need more documents" return { 'answer': answer.response, 'confidence': answer.confidence_score, 'status': status, 'sources': answer.sources } # Usage result = get_verified_answer(hippo, chat_id, "What is the SLA?") print(f"{result['status']}") print(f"Answer: {result['answer']}") ``` ### Use Source Citations ```python theme={null} def display_answer_with_sources(answer): """Show answer with full source attribution""" print(f"Answer: {answer.response}\n") print(f"Confidence: {answer.confidence_score:.2f}\n") if answer.sources: print(f"Sources ({len(answer.sources)}):") for i, source in enumerate(answer.sources, 1): print(f"{i}. {source.file_name} (Page {source.page_number})") print(f" Relevance: {source.relevance_score:.2f}") print(f" Excerpt: {source.excerpt[:100]}...\n") else: print("⚠️ No sources found - answer may be uncertain") # Usage answer = hippo.submit_ask(chat_id, question) display_answer_with_sources(answer) ``` ## Maintenance & Monitoring ### Regular Cleanup ```python theme={null} def cleanup_workspace(hippo): """Clean up old/unused resources""" folders = hippo.get_folders() for folder in folders: # Delete empty folders if folder.file_count == 0: print(f"Deleting empty folder: {folder.name}") hippo.delete_folder(folder.id) continue # Clean up old test chats chats = hippo.get_chats(folder.id) for chat in chats: if "test" in chat.name.lower() and chat.message_count == 0: print(f"Deleting test chat: {chat.name}") hippo.delete_chat(chat.id) # Run monthly cleanup_workspace(hippo) ``` ### Monitor Usage ```python theme={null} from cerevox import Account account = Account(api_key="your-api-key") # Check usage usage = account.get_usage() print(f"API Calls: {usage.total_requests}") print(f"Documents Processed: {usage.documents_processed}") print(f"Questions Asked: {usage.questions_asked}") # Check if approaching limits if usage.total_requests > usage.rate_limit * 0.8: print("⚠️ Approaching rate limit - consider upgrading") ``` ## Production Checklist * [ ] Use environment variables for API keys * [ ] Never commit API keys to version control * [ ] Implement user-specific chat isolation * [ ] Delete sensitive data when no longer needed * [ ] Review uploaded documents for PII/sensitive data * [ ] Use async API for production workloads * [ ] Implement connection pooling * [ ] Add retry logic for failed requests * [ ] Cache frequently asked questions if appropriate * [ ] Monitor response times ```python theme={null} from cerevox import HippoError try: answer = hippo.submit_ask(chat_id, question) except HippoError as e: if "rate limit" in str(e).lower(): # Handle rate limiting time.sleep(60) answer = hippo.submit_ask(chat_id, question) elif "not found" in str(e).lower(): # Handle missing resources print(f"Error: Chat or folder not found") else: # Log and handle other errors logger.error(f"Hippo error: {e}") ``` * [ ] Track answer confidence scores * [ ] Monitor API usage and costs * [ ] Log low-confidence answers for review * [ ] Set up alerts for errors * [ ] Review source citations quality * [ ] Document folder organization strategy * [ ] Keep inventory of uploaded documents * [ ] Document common questions and answers * [ ] Maintain change log for document updates * [ ] Create runbooks for common operations ## Common Pitfalls to Avoid **Don't**: * Mix unrelated documents in one folder * Use vague question phrasing * Ignore confidence scores * Upload scanned PDFs without OCR * Create new chats for every question * Forget to clean up test resources * Share API keys or commit them to git **Do**: * Group related documents logically * Ask specific, clear questions * Verify low-confidence answers with sources * Use text-based documents when possible * Reuse chats for related conversations * Regularly clean up unused resources * Use environment variables for API keys ## Complete Production Example ```python theme={null} import os import asyncio import logging from cerevox import AsyncHippo, HippoError logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) class ProductionRAGSystem: def __init__(self): self.hippo = None self.folders = {} self.chats = {} async def setup(self): """Initialize production RAG system""" api_key = os.getenv("CEREVOX_API_KEY") if not api_key: raise ValueError("CEREVOX_API_KEY not set") self.hippo = AsyncHippo(api_key=api_key) # Create knowledge bases self.folders['support'] = await self.hippo.create_folder( "Customer Support KB", "Support docs, FAQs, troubleshooting" ) logger.info(f"Created folder: {self.folders['support'].name}") async def upload_documents(self, folder_key, file_paths): """Batch upload with error handling""" folder_id = self.folders[folder_key].id tasks = [] for path in file_paths: if os.path.exists(path): tasks.append(self.hippo.upload_file(folder_id, path)) else: logger.warning(f"File not found: {path}") try: files = await asyncio.gather(*tasks, return_exceptions=True) successful = [f for f in files if not isinstance(f, Exception)] failed = [f for f in files if isinstance(f, Exception)] logger.info(f"Uploaded {len(successful)} files") if failed: logger.error(f"Failed uploads: {len(failed)}") return successful except Exception as e: logger.error(f"Upload error: {e}") return [] async def ask_question(self, folder_key, question): """Ask with retry logic and validation""" # Get or create chat if folder_key not in self.chats: folder_id = self.folders[folder_key].id self.chats[folder_key] = await self.hippo.create_chat( folder_id, f"{folder_key.title()} Chat" ) chat_id = self.chats[folder_key].id # Ask with retry max_retries = 3 for attempt in range(max_retries): try: answer = await self.hippo.submit_ask(chat_id, question) # Log quality metrics logger.info( f"Q&A - Confidence: {answer.confidence_score:.2f}, " f"Sources: {len(answer.sources)}" ) return { 'answer': answer.response, 'confidence': answer.confidence_score, 'sources': answer.sources, 'verified': answer.confidence_score >= 0.7 } except HippoError as e: if attempt < max_retries - 1: await asyncio.sleep(2 ** attempt) # Exponential backoff continue else: logger.error(f"Failed after {max_retries} attempts: {e}") raise async def cleanup(self): """Clean up resources""" if self.hippo: await self.hippo.close() # Usage async def main(): system = ProductionRAGSystem() try: await system.setup() # Upload docs docs = ["faq.pdf", "guide.pdf", "troubleshooting.pdf"] await system.upload_documents('support', docs) # Ask questions result = await system.ask_question( 'support', "How do I reset my password?" ) print(f"Answer: {result['answer']}") print(f"Verified: {result['verified']}") finally: await system.cleanup() # Run asyncio.run(main()) ``` ## Next Steps Production-ready code examples Advanced optimization techniques # Chat Sessions Source: https://docs.cerevox.ai/hippo/chat Create and manage conversation contexts for Q&A # Chat Sessions Chat sessions provide conversation context for asking questions about your documents. ## What are Chat Sessions? **Chat sessions** are conversation contexts linked to a folder: * Each chat is connected to one folder * Maintains conversation history for follow-up questions * Multiple chats can be created per folder * Isolates different conversation topics or users Think of chats as **conversation threads** - each one remembers previous questions and answers within that thread. ## Core Operations ### Create a Chat ```python Sync theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key") # Create chat linked to a folder chat = hippo.create_chat( folder_id="folder_123", chat_name="Technical Support Q&A" ) print(f"Created: {chat.name}") print(f"Chat ID: {chat.id}") ``` ```python Async theme={null} from cerevox import AsyncHippo async with AsyncHippo(api_key="your-api-key") as hippo: chat = await hippo.create_chat( folder_id="folder_123", chat_name="Technical Support" ) print(f"Chat ID: {chat.id}") ``` **Response fields:** * `id`: Unique chat identifier * `name`: Chat name * `folder_id`: Associated folder * `created_at`: Creation timestamp * `message_count`: Number of Q\&A exchanges ### List Chats ```python Get All Chats in Folder theme={null} # Get all chats for a folder chats = hippo.get_chats(folder_id="folder_123") for chat in chats: print(f"{chat.name}: {chat.message_count} messages") ``` ```python Async theme={null} chats = await hippo.get_chats(folder_id="folder_123") for chat in chats: print(f"{chat.name} - {chat.message_count} Q&A") ``` ### Get Chat Details ```python Sync theme={null} # Get specific chat information chat = hippo.get_chat(chat_id="chat_456") print(f"Name: {chat.name}") print(f"Folder: {chat.folder_id}") print(f"Messages: {chat.message_count}") print(f"Created: {chat.created_at}") ``` ```python Async theme={null} chat = await hippo.get_chat(chat_id="chat_456") ``` ### Update Chat Name ```python Sync theme={null} # Rename a chat updated_chat = hippo.update_chat( chat_id=chat.id, chat_name="Customer Support - Priority" ) print(f"Renamed to: {updated_chat.name}") ``` ```python Async theme={null} updated = await hippo.update_chat( chat_id=chat.id, chat_name="New Name" ) ``` ### Delete Chat ```python Sync theme={null} # Delete chat and its Q&A history hippo.delete_chat(chat_id="chat_456") print("Chat deleted successfully") ``` ```python Async theme={null} await hippo.delete_chat(chat_id="chat_456") ``` Deleting a chat removes all Q\&A history. The folder and files remain intact. ## Chat Organization Patterns Create individual chats for each user: ```python theme={null} # Create chat for each customer customer_chat = hippo.create_chat( folder_id=support_docs_folder.id, chat_name=f"Support - Customer {customer_id}" ) ``` **Benefit**: Personalized conversation history per user Organize by conversation topic: ```python theme={null} topics = ["Authentication", "Billing", "API Usage", "Troubleshooting"] chats = {} for topic in topics: chat = hippo.create_chat(folder.id, f"Support - {topic}") chats[topic] = chat ``` **Benefit**: Clear separation of conversation topics Create temporary chats for sessions: ```python theme={null} # Create chat for this user session session_chat = hippo.create_chat( folder.id, f"Session {session_id} - {datetime.now()}" ) # ... ask questions ... # Clean up after session ends hippo.delete_chat(session_chat.id) ``` **Benefit**: Automatic cleanup of temporary conversations Chats for different teams accessing same docs: ```python theme={null} departments = ["Sales", "Support", "Engineering"] for dept in departments: chat = hippo.create_chat( product_docs_folder.id, f"{dept} Team Chat" ) ``` **Benefit**: Department-specific conversation tracking ## Conversation Context Chat sessions maintain context for follow-up questions: ```python theme={null} # First question answer1 = hippo.submit_ask( chat.id, "What is the API rate limit?" ) # Answer: "The API rate limit is 1000 requests per hour..." # Follow-up question (chat remembers previous context) answer2 = hippo.submit_ask( chat.id, "How can I increase it?" # "it" refers to rate limit from Q1 ) # Answer: "To increase your rate limit, contact support..." # Another follow-up answer3 = hippo.submit_ask( chat.id, "What's the process for that?" # "that" refers to Q2 ) # Answer: "The process for increasing rate limits is..." ``` **Context window**: Chats remember the last 10 Q\&A exchanges for context. ## View Chat History Get all questions and answers from a chat: ```python Sync theme={null} # Get full Q&A history asks = hippo.get_asks(chat_id=chat.id) print(f"Chat history ({len(asks)} Q&A):\n") for i, ask in enumerate(asks, 1): print(f"{i}. Q: {ask.question}") print(f" A: {ask.response}") print(f" Sources: {len(ask.sources)}\n") ``` ```python Async theme={null} asks = await hippo.get_asks(chat_id=chat.id) for ask in asks: print(f"Q: {ask.question}") print(f"A: {ask.response}\n") ``` ## Complete Example: Multi-User Support System ```python theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key") # 1. Create knowledge base support_folder = hippo.create_folder( "Customer Support Docs", "FAQs, troubleshooting, and product guides" ) # 2. Upload support documents docs = [ "faq.pdf", "troubleshooting-guide.pdf", "product-manual.pdf" ] for doc in docs: hippo.upload_file(support_folder.id, doc) # 3. Create chats for different purposes chats = {} # General support chat chats['general'] = hippo.create_chat( support_folder.id, "General Support" ) # Technical support chat chats['technical'] = hippo.create_chat( support_folder.id, "Technical Support" ) # Billing inquiries chat chats['billing'] = hippo.create_chat( support_folder.id, "Billing Support" ) # 4. Route questions to appropriate chat def ask_support(question, category='general'): chat_id = chats[category].id answer = hippo.submit_ask(chat_id, question) print(f"Category: {category}") print(f"Q: {question}") print(f"A: {answer.response}\n") # Use the system ask_support("How do I reset my password?", "general") ask_support("Why is the API returning 500 errors?", "technical") ask_support("How do I update my payment method?", "billing") ``` ## Best Practices **Good**: "Customer Support - Authentication Issues" **Bad**: "Chat 1", "Test", "Untitled" Clear names help identify chats when you have many. ```python theme={null} # Delete old or test chats chats = hippo.get_chats(folder_id) for chat in chats: if "test" in chat.name.lower() or chat.message_count == 0: hippo.delete_chat(chat.id) print(f"Deleted: {chat.name}") ``` Keep your workspace organized by removing unused chats. **Don't** share chats across users: ```python theme={null} # Bad: Shared chat for all users shared_chat = hippo.create_chat(folder.id, "Everyone") # Good: Individual chats per user user_chat = hippo.create_chat( folder.id, f"User {user_id}" ) ``` Individual chats prevent context confusion and protect privacy. **Temporary chats** (delete after session): ```python theme={null} chat = hippo.create_chat(folder.id, f"Session {session_id}") # ... use chat ... hippo.delete_chat(chat.id) ``` **Permanent chats** (keep for history): ```python theme={null} chat = hippo.create_chat(folder.id, f"Customer {customer_id}") # Keep for entire customer relationship ``` ## Limits & Quotas **Free tier**: Up to 50 chats **Pro tier**: Up to 1,000 chats **Enterprise**: Unlimited **All tiers**: Unlimited Q\&A exchanges **Context window**: Last 10 exchanges used for context ## Next Steps Ask questions in your chats Optimize chat organization Complete RAG workflow examples # File Operations Source: https://docs.cerevox.ai/hippo/files Upload and manage documents in Hippo folders # File Operations Upload and manage documents that power your RAG Q\&A system. ## Upload Files ### Upload from Local File ```python Sync theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key") # Upload a local file file = hippo.upload_file( folder_id="folder_123", file_path="documents/user-guide.pdf" ) print(f"Uploaded: {file.name}") print(f"File ID: {file.id}") print(f"Status: {file.status}") ``` ```python Async theme={null} from cerevox import AsyncHippo async with AsyncHippo(api_key="your-api-key") as hippo: file = await hippo.upload_file( folder_id="folder_123", file_path="documents/user-guide.pdf" ) print(f"Uploaded: {file.name}") ``` ### Upload from URL ```python Sync theme={null} # Upload directly from a URL file = hippo.upload_file_from_url( folder_id="folder_123", file_url="https://example.com/whitepaper.pdf", file_name="whitepaper.pdf" # Optional custom name ) print(f"Uploaded from URL: {file.name}") ``` ```python Async theme={null} file = await hippo.upload_file_from_url( folder_id="folder_123", file_url="https://example.com/document.pdf", file_name="document.pdf" ) ``` ### Batch Upload ```python Sync - Sequential theme={null} files = [] for file_path in ["doc1.pdf", "doc2.docx", "doc3.pptx"]: file = hippo.upload_file(folder_id, file_path) files.append(file) print(f"Uploaded: {file.name}") ``` ```python Async - Concurrent theme={null} import asyncio async with AsyncHippo() as hippo: # Upload multiple files concurrently upload_tasks = [ hippo.upload_file(folder_id, "doc1.pdf"), hippo.upload_file(folder_id, "doc2.docx"), hippo.upload_file(folder_id, "doc3.pptx") ] files = await asyncio.gather(*upload_tasks) print(f"Uploaded {len(files)} files concurrently") ``` ## Supported File Formats * PDF (.pdf) * Word (.docx, .doc) * PowerPoint (.pptx, .ppt) * Text (.txt) * RTF (.rtf) * Excel (.xlsx, .xls) * CSV (.csv) * TSV (.tsv) * HTML (.html) * MHTML (.mhtml) * Markdown (.md) **File size limits:** * Max file size: 100MB per file * Contact support for larger files or custom formats ## List Files ```python Sync theme={null} # Get all files in a folder files = hippo.get_files(folder_id="folder_123") for file in files: print(f"{file.name} - {file.status} - {file.size_bytes} bytes") ``` ```python Async theme={null} files = await hippo.get_files(folder_id="folder_123") for file in files: print(f"{file.name}: {file.status}") ``` **File status values:** * `uploading`: File is being uploaded * `processing`: File is being indexed * `completed`: File is ready for Q\&A * `failed`: Processing failed ## Get File Details ```python Sync theme={null} # Get specific file information file = hippo.get_file(file_id="file_456") print(f"Name: {file.name}") print(f"Status: {file.status}") print(f"Size: {file.size_bytes} bytes") print(f"Pages: {file.page_count}") print(f"Uploaded: {file.created_at}") ``` ```python Async theme={null} file = await hippo.get_file(file_id="file_456") ``` ## Delete Files ```python Sync theme={null} # Delete a file hippo.delete_file(file_id="file_456") print("File deleted successfully") ``` ```python Async theme={null} await hippo.delete_file(file_id="file_456") ``` Deleted files cannot be recovered. The file will be removed from all chats and answers that referenced it. ## File Processing ### Processing Time Files are automatically processed after upload: File is uploaded to Cerevox (a few seconds) Document is parsed for text and structure (10s - 2min) Content is split into semantic chunks (a few seconds) Chunks are indexed for search (10s - 1min) File is ready for Q\&A! **Total processing time:** * Small files (\< 10 pages): 10-30 seconds * Medium files (10-100 pages): 30-120 seconds * Large files (> 100 pages): 2-5 minutes ### Monitor Processing Status ```python theme={null} import time # Upload file file = hippo.upload_file(folder_id, "large-document.pdf") # Poll until processing completes while file.status != "completed": time.sleep(5) file = hippo.get_file(file.id) print(f"Status: {file.status}") print("File ready for Q&A!") ``` ## Best Practices **Before uploading:** * Ensure PDFs are text-based (not scanned images) * Check that documents aren't password-protected * Verify file isn't corrupted **Tip**: OCR (scanned) PDFs work but may have lower accuracy. Use text-based PDFs when possible. **Reduce processing time:** * Remove unnecessary pages (covers, blanks, ads) * Compress images in PDFs * Split very large documents (500+ pages) Smaller, focused documents = faster processing + better search results ```python theme={null} # Good: Upload product docs together folder = hippo.create_folder("Product V2 Docs") docs = ["overview.pdf", "features.pdf", "api.pdf"] for doc in docs: hippo.upload_file(folder.id, doc) ``` Related documents in the same folder enable better cross-document Q\&A. **Good**: `product-api-authentication-guide.pdf` **Bad**: `doc1.pdf`, `untitled.pdf` Descriptive names help with source citations and debugging. ```python theme={null} # Async = 10x faster for multiple files async with AsyncHippo() as hippo: tasks = [hippo.upload_file(folder_id, f) for f in files] results = await asyncio.gather(*tasks) ``` Async concurrent uploads are significantly faster than sequential. ## Complete Example: Batch Upload ```python theme={null} import asyncio from pathlib import Path from cerevox import AsyncHippo async def batch_upload_directory(folder_id, directory_path): """Upload all PDFs from a directory""" async with AsyncHippo(api_key="your-api-key") as hippo: # Get all PDF files pdf_files = list(Path(directory_path).glob("*.pdf")) print(f"Found {len(pdf_files)} PDF files") # Upload concurrently tasks = [ hippo.upload_file(folder_id, str(pdf)) for pdf in pdf_files ] files = await asyncio.gather(*tasks) # Report results print(f"\n✅ Uploaded {len(files)} files:") for file in files: print(f" - {file.name} ({file.status})") return files # Usage folder = await hippo.create_folder("Uploaded Docs") files = await batch_upload_directory(folder.id, "./documents") ``` ## Error Handling ```python theme={null} from cerevox import Hippo, HippoError hippo = Hippo() try: file = hippo.upload_file(folder_id, "document.pdf") print(f"Uploaded: {file.name}") except FileNotFoundError: print("Error: File not found") except HippoError as e: if "unsupported format" in str(e).lower(): print("Error: File format not supported") elif "too large" in str(e).lower(): print("Error: File exceeds size limit") else: print(f"Error: {e}") ``` ## Next Steps Create chats to ask questions Ask questions over uploaded files Optimize file preparation # Folder Management Source: https://docs.cerevox.ai/hippo/folders Organize documents into searchable knowledge bases # Folder Management Folders are the foundation of Hippo's RAG system - they organize your documents into isolated, searchable knowledge bases. ## What are Folders? Think of folders as **knowledge base containers**: * Each folder holds a collection of related documents * Documents are automatically indexed for semantic search * Chats are linked to folders to access their documents * Folders provide logical separation of knowledge domains **Best practice**: Create separate folders for different knowledge domains (e.g., "Product Docs", "HR Policies", "Customer Support") ## Core Operations ### Create a Folder ```python Sync theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key") # Create folder with name and description folder = hippo.create_folder( name="Product Documentation", description="User guides, API docs, and tutorials" ) print(f"Created: {folder.name}") print(f"ID: {folder.id}") ``` ```python Async theme={null} from cerevox import AsyncHippo async with AsyncHippo(api_key="your-api-key") as hippo: folder = await hippo.create_folder( name="Product Documentation", description="User guides and API docs" ) print(f"Created: {folder.name}") ``` **Response fields:** * `id`: Unique folder identifier * `name`: Folder name * `description`: Folder description * `created_at`: Creation timestamp * `file_count`: Number of files in folder ### List All Folders ```python Sync theme={null} # Get all your folders folders = hippo.get_folders() for folder in folders: print(f"{folder.name}: {folder.file_count} files") ``` ```python Async theme={null} folders = await hippo.get_folders() for folder in folders: print(f"{folder.name}: {folder.file_count} files") ``` ### Get Folder Details ```python Sync theme={null} # Get specific folder by ID folder = hippo.get_folder(folder_id="folder_123") print(f"Name: {folder.name}") print(f"Files: {folder.file_count}") print(f"Created: {folder.created_at}") ``` ```python Async theme={null} folder = await hippo.get_folder(folder_id="folder_123") ``` ### Update Folder ```python Sync theme={null} # Update folder name or description updated_folder = hippo.update_folder( folder_id=folder.id, name="Updated Product Docs", description="Complete product documentation library" ) print(f"Updated: {updated_folder.name}") ``` ```python Async theme={null} updated = await hippo.update_folder( folder_id=folder.id, name="Updated Name" ) ``` ### Delete Folder **Deleting a folder is permanent** and removes all files, chats, and Q\&A history within it! ```python Sync theme={null} # Delete folder and all its contents hippo.delete_folder(folder_id=folder.id) print("Folder deleted successfully") ``` ```python Async theme={null} await hippo.delete_folder(folder_id=folder.id) ``` ## Folder Organization Strategies Create folders for each department or team: * "Engineering Docs" * "Sales Playbooks" * "HR Policies" * "Customer Support" **Benefit**: Easy access control and logical separation Organize by product lines: * "Product A Documentation" * "Product B User Guides" * "Service X Training Materials" **Benefit**: Clear product-specific knowledge bases Useful for financial or periodic documents: * "Q1 2025 Reports" * "2024 Annual Filings" * "Monthly Updates - March 2025" **Benefit**: Easy to archive and reference historical data For client-specific or project-based work: * "Client: Acme Corp" * "Project: Website Redesign" * "Customer: TechStartup Inc" **Benefit**: Isolated knowledge per client/project ## Folder Limits & Quotas **Free tier**: Up to 100 files **Pro tier**: Up to 10,000 files **Enterprise**: Unlimited Contact sales for custom limits **Free tier**: Up to 10 folders **Pro tier**: Up to 100 folders **Enterprise**: Unlimited See [pricing](https://cerevox.ai/pricing) ## Best Practices **Good**: "Product API Documentation v2.0" **Bad**: "Docs", "Folder1", "Misc" Descriptive names make it easier to manage multiple folders and understand their purpose at a glance. ```python theme={null} folder = hippo.create_folder( name="Customer Support KB", description="FAQs, troubleshooting guides, and product manuals for support team" ) ``` Descriptions help team members understand folder contents without exploring files. ```python theme={null} # Delete old or unused folders folders = hippo.get_folders() for folder in folders: if folder.file_count == 0: print(f"Empty folder: {folder.name}") # Optionally delete # hippo.delete_folder(folder.id) ``` Clean up empty or outdated folders to keep your workspace organized. **Don't**: Create 50 folders with 2-3 files each **Do**: Create fewer folders with more related documents Folders with more documents generally yield better search results. Start with broad categories and refine as your knowledge base grows: 1. Start: "Product Documentation" 2. As it grows → Split into "Product A Docs", "Product B Docs" 3. As it grows more → Split by version or feature Avoid premature over-organization. ## Complete Example: Manage Product Docs ```python theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key") # 1. Create folders for different product areas folders = {} for area in ["User Guides", "API Reference", "Tutorials", "Release Notes"]: folder = hippo.create_folder( name=f"Product Docs - {area}", description=f"{area} for all products" ) folders[area] = folder print(f"Created: {folder.name}") # 2. Upload files to appropriate folders hippo.upload_file(folders["User Guides"].id, "getting-started.pdf") hippo.upload_file(folders["API Reference"].id, "api-v2.pdf") hippo.upload_file(folders["Tutorials"].id, "tutorial-basics.pdf") # 3. List all folders and their stats all_folders = hippo.get_folders() print("\nFolder Summary:") for folder in all_folders: print(f" {folder.name}: {folder.file_count} files") # 4. Clean up if needed # hippo.delete_folder(folder_id=folders["Release Notes"].id) ``` ## Next Steps Learn to upload and manage files Create chats linked to folders Optimize folder organization # Hippo - RAG & Retrieval Source: https://docs.cerevox.ai/hippo/overview AI-powered semantic search and Q&A with 80% cost reduction # Hippo: RAG & Retrieval 🦛 **Build intelligent Q\&A systems over documents with flagship accuracy at mini model cost.** Hippo delivers precision retrieval with 70% smaller context windows, achieving **80% cost reduction** while maintaining **99.5% accuracy match** to flagship models. **80% COST REDUCTION** Only retrieve relevant chunks, not entire documents **99.5% ACCURACY MATCH** Flagship model quality with intelligent retrieval **70% SMALLER CONTEXT** Precision RAG eliminates noise, keeps what matters ## What is Hippo? Hippo is Cerevox's **RAG (Retrieval-Augmented Generation) API** that enables AI agents to search and query document collections with natural language. Instead of sending entire documents to your LLM (expensive, slow, noisy), Hippo: 1. **Indexes** your documents with semantic understanding 2. **Retrieves** only the most relevant chunks (70% smaller context) 3. **Generates** AI answers with source citations 4. **Saves** you 80% on LLM costs while matching flagship accuracy **Perfect for**: Customer support bots, internal knowledge bases, document Q\&A, research assistants, and any AI system that needs to "know" information from documents. ## Core Concepts **Folders** are collections of documents that form a searchable knowledge base. * Each folder is an isolated knowledge domain * Upload PDFs, DOCX, PPTX, and more * Automatically indexed for semantic search * Support 1 to 10,000+ documents per folder **Use cases**: Product docs, customer records, research papers, legal cases **Files** are the documents you upload to folders. * Support 12+ formats: PDF, DOCX, PPTX, XLSX, TXT, HTML, CSV, etc. * Upload from local files or URLs * Automatic processing and indexing * Rich metadata extraction **Processing**: Files are automatically parsed, chunked, and indexed for retrieval **Chat sessions** maintain conversation context for Q\&A. * Each chat is connected to a folder * Maintains conversation history * Supports follow-up questions * Multiple chats per folder **Use cases**: Support conversations, research sessions, document analysis **Asks** are questions submitted to a chat that generate AI-powered answers. * Natural language questions * AI-generated answers with source citations * Confidence scores for each answer * Full conversation history accessible **Returns**: Answer text + source documents + page numbers + confidence scores ## How It Works ```mermaid theme={null} graph LR A[Your Documents] --> B[Upload to Folder] B --> C[Automatic Indexing] C --> D[Create Chat Session] D --> E[Ask Questions] E --> F[Precision Retrieval
70% smaller context] F --> G[AI Answer + Citations] G --> H[80% Cost Savings] ``` Organize documents into a knowledge base Add documents from local files or URLs Start a conversation session linked to the folder Submit natural language questions and get AI answers with sources ## Quick Example ```python Sync API theme={null} from cerevox import Hippo # Initialize hippo = Hippo(api_key="your-api-key") # 1. Create knowledge base folder = hippo.create_folder("Product Documentation") # 2. Upload documents hippo.upload_file(folder.id, "user-guide.pdf") hippo.upload_file_from_url(folder.id, "https://example.com/api-docs.pdf") # 3. Create chat chat = hippo.create_chat(folder.id, "Support Q&A") # 4. Ask questions answer = hippo.submit_ask( chat.id, "How do I authenticate users?" ) print(f"Answer: {answer.response}") print(f"Sources: {[s.file_name for s in answer.sources]}") # 80% cost reduction vs. full document retrieval! ``` ```python Async API theme={null} import asyncio from cerevox import AsyncHippo async def main(): async with AsyncHippo(api_key="your-api-key") as hippo: # Create and upload concurrently folder = await hippo.create_folder("Product Docs") files = await asyncio.gather( hippo.upload_file(folder.id, "guide.pdf"), hippo.upload_file(folder.id, "docs.pdf") ) # Create chat and ask chat = await hippo.create_chat(folder.id, "Q&A") answer = await hippo.submit_ask(chat.id, "How do I get started?") print(f"Answer: {answer.response}") asyncio.run(main()) ``` ## Key Features **AI-powered understanding** * Finds relevant content by meaning, not just keywords * Handles synonyms and context * Multi-language support **Verify every answer** * Exact source documents * Page numbers included * Confidence scores **Contextual follow-ups** * Chats remember previous questions * Support clarifying questions * Full history accessible **12+ file formats** * PDF, DOCX, PPTX, XLSX * TXT, HTML, CSV, and more * Automatic format detection **High performance** * Full async/await support * Concurrent uploads * Batch processing **Production proven** * Automatic retries * Error handling * Usage tracking ## The Cost Savings Advantage ```python theme={null} # Traditional approach: Send entire documents documents = load_all_documents() # Large context context = "\n\n".join([doc.content for doc in documents]) # Send to LLM - EXPENSIVE llm_response = openai.chat.completions.create( messages=[{ "role": "user", "content": f"Context: {context}\n\nQuestion: {question}" }], model="gpt-4" # Flagship model required ) # High token costs: 10,000+ tokens per query # Slow: Large context = slower processing # Noisy: Irrelevant content confuses the model ``` ```python theme={null} # Hippo approach: Precision retrieval answer = hippo.submit_ask(chat_id, question) # BEHIND THE SCENES: # 1. Semantic search finds relevant chunks only # 2. 70% smaller context (3,000 tokens vs 10,000) # 3. Same accuracy as flagship models # 4. Source citations included print(f"Answer: {answer.response}") print(f"Sources: {answer.sources}") # 80% COST REDUCTION # Faster: Smaller context = faster responses # Cleaner: Only relevant content # Verified: Source citations for every answer ``` ## Use Cases **Build AI support agents that answer customer questions** * Upload help docs, FAQs, and knowledge base * Customers ask questions in natural language * Get instant answers with source citations * 80% reduction in support costs Example: "How do I reset my password?" → Answer + link to help article **Make company knowledge searchable** * Upload policies, procedures, onboarding docs * Employees ask questions, get instant answers * Reduce time spent searching for information * Keep knowledge always accessible Example: "What's our remote work policy?" → Answer from HR handbook **Search contracts and legal documents** * Upload contracts, agreements, legal cases * Ask questions about terms, clauses, precedents * Get answers with exact citations * Verify every response with sources Example: "What are the termination clauses?" → Answer with contract references **Query research papers and technical docs** * Upload papers, reports, technical documentation * Ask research questions * Get synthesized answers from multiple sources * Citations to original papers Example: "What methods did Smith et al. use?" → Answer from relevant papers **Query financial reports and filings** * Upload 10-Ks, earnings reports, analyst notes * Ask about metrics, trends, risks * Get answers with exact page references * Compare across multiple documents Example: "What were Q3 revenue drivers?" → Answer from earnings call ## Hippo vs. Traditional RAG | Feature | Traditional RAG | Hippo RAG | | -------------------- | --------------------------------------------- | ----------------------------------- | | **Context Size** | Full documents (10,000+ tokens) | Relevant chunks only (3,000 tokens) | | **Cost per Query** | $0.10 - $0.50 | $0.02 - $0.10 (80% reduction) | | **Accuracy** | Good (with flagship models) | 99.5% match (with mini models) | | **Response Time** | Slow (large context) | Fast (smaller context) | | **Source Citations** | Manual implementation | Built-in with confidence scores | | **Setup Complexity** | High (vector DB, embeddings, retrieval logic) | Low (API-only, no infrastructure) | | **Maintenance** | Ongoing (infrastructure, tuning) | None (managed service) | ## API Clients Hippo provides both synchronous and asynchronous clients: ```python Synchronous theme={null} from cerevox import Hippo # Best for: Simple scripts, notebooks, learning hippo = Hippo(api_key="your-api-key") folder = hippo.create_folder("Docs") chat = hippo.create_chat(folder.id) answer = hippo.submit_ask(chat.id, "Question?") ``` ```python Asynchronous theme={null} from cerevox import AsyncHippo import asyncio # Best for: Production apps, high throughput, web servers async def main(): async with AsyncHippo(api_key="your-api-key") as hippo: folder = await hippo.create_folder("Docs") chat = await hippo.create_chat(folder.id) answer = await hippo.submit_ask(chat.id, "Question?") asyncio.run(main()) ``` ## Next Steps Build your first Q\&A system in 5 minutes Organize documents effectively Upload and manage documents Create conversation contexts Ask questions and get answers Optimize retrieval quality and costs *** **Ready to save 80%?** Check out the [quickstart guide](/hippo/quickstart) or explore [RAG examples](/examples/rag-workflow). # Q&A System Source: https://docs.cerevox.ai/hippo/questions Ask questions and get AI-powered answers with citations # Q\&A System Submit questions to your documents and get AI-generated answers with source citations and **80% cost savings**. ## How it Works When you ask a question, Hippo: Finds relevant chunks from your documents (not keyword matching) Retrieves only what's needed (**70% smaller context**) Generates answer using mini model with flagship accuracy Returns answer + source documents + confidence scores **Result**: 99.5% accuracy match to flagship models at 80% lower cost! ## Submit Questions ### Basic Q\&A ```python Sync theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key") # Ask a question answer = hippo.submit_ask( chat_id="chat_123", question="How do I authenticate users in the API?" ) # Access the response print(f"Answer: {answer.response}") print(f"Confidence: {answer.confidence_score}") print(f"Sources: {len(answer.sources)}") ``` ```python Async theme={null} from cerevox import AsyncHippo async with AsyncHippo() as hippo: answer = await hippo.submit_ask( chat_id="chat_123", question="What is the API rate limit?" ) print(f"Answer: {answer.response}") ``` ### Response Structure The answer object contains: The AI-generated answer to your question The original question (as processed) Confidence score from 0.0 to 1.0 * `0.9+`: High confidence * `0.7-0.9`: Medium confidence * `< 0.7`: Low confidence (verify sources) List of source documents cited in the answer Each source contains: * `file_name`: Name of the source file * `file_id`: Unique file identifier * `page_number`: Page where info was found * `relevance_score`: How relevant (0.0-1.0) * `excerpt`: Text snippet from source When the question was answered ### Access Source Citations ```python theme={null} # Ask question answer = hippo.submit_ask(chat_id, "What is the refund policy?") # Display answer print(f"Answer: {answer.response}\n") # Show sources print(f"Based on {len(answer.sources)} sources:") for source in answer.sources: print(f"\n📄 {source.file_name}") print(f" Page: {source.page_number}") print(f" Relevance: {source.relevance_score:.2f}") print(f" Excerpt: {source.excerpt[:100]}...") ``` ## Question History ### Get All Q\&A for a Chat ```python Sync theme={null} # Get complete Q&A history asks = hippo.get_asks(chat_id="chat_123") for ask in asks: print(f"Q: {ask.question}") print(f"A: {ask.response}") print(f"Confidence: {ask.confidence_score}\n") ``` ```python Async theme={null} asks = await hippo.get_asks(chat_id="chat_123") for ask in asks: print(f"{ask.question} → {ask.response[:100]}...") ``` ### Get Specific Q\&A by Index ```python theme={null} # Get the 3rd question/answer from chat ask = hippo.get_ask_by_index( chat_id="chat_123", index=2 # 0-indexed ) print(f"Q: {ask.question}") print(f"A: {ask.response}") ``` ## Follow-up Questions Chats remember context for follow-up questions: ```python theme={null} # First question answer1 = hippo.submit_ask( chat_id, "What are the API authentication methods?" ) print(f"A1: {answer1.response}") # Follow-up (references "methods" from Q1) answer2 = hippo.submit_ask( chat_id, "Which one is most secure?" # Understands context ) print(f"A2: {answer2.response}") # Another follow-up (references "secure method" from Q2) answer3 = hippo.submit_ask( chat_id, "How do I implement it?" # Knows what "it" refers to ) print(f"A3: {answer3.response}") ``` **Context memory**: Last 10 Q\&A exchanges are used for context in follow-up questions. ## Question Types **Best for**: Specific information retrieval ```python theme={null} questions = [ "What is the API rate limit?", "What's the refund policy?", "What programming languages are supported?" ] ``` **Performance**: Highest accuracy (99.5%+ with clear documentation) **Best for**: Step-by-step instructions ```python theme={null} questions = [ "How do I reset my password?", "How can I integrate with Stripe?", "How do I export data to CSV?" ] ``` **Performance**: Excellent when docs contain clear procedures **Best for**: Comparing options or features ```python theme={null} questions = [ "What's the difference between Basic and Pro plans?", "How does REST compare to GraphQL in our API?", "Which authentication method is most secure?" ] ``` **Performance**: Good when comparison info exists in docs **Best for**: Synthesizing information ```python theme={null} questions = [ "What are the main features of Product X?", "Summarize the key risks in this contract", "What were the Q3 revenue highlights?" ] ``` **Performance**: Good for overview questions across multiple sources **Best for**: Quick verification ```python theme={null} questions = [ "Does the API support webhooks?", "Can I export data to Excel?", "Is there a mobile app?" ] ``` **Performance**: Excellent with clear documentation ## Writing Effective Questions ✅ "What is the API rate limit for Pro plan users?" ❌ "Tell me about limits" Specific questions get specific answers ✅ "How do I reset my password?" ❌ "password reset procedure documentation" Write questions as you'd ask a person ✅ "What is the refund policy?" ❌ "What's the refund policy and cancellation process and payment methods?" Multiple questions → Ask separately for better answers ✅ "In the REST API, how do I authenticate?" ❌ "How authenticate?" (ambiguous) Context helps when docs cover multiple systems ## Interpreting Confidence Scores ```python theme={null} answer = hippo.submit_ask(chat_id, question) if answer.confidence_score >= 0.9: print("✅ High confidence - Answer is very reliable") elif answer.confidence_score >= 0.7: print("⚠️ Medium confidence - Check sources to verify") else: print("⚠️ Low confidence - Information may not be in documents") print("Consider:") print("- Rephrasing the question") print("- Adding relevant documents") print("- Checking if info exists in uploaded files") ``` ## Complete Example: Support Bot ```python theme={null} from cerevox import Hippo class SupportBot: def __init__(self, api_key): self.hippo = Hippo(api_key=api_key) self.folder = None self.chat = None def setup(self, support_docs): """Initialize knowledge base""" # Create folder self.folder = self.hippo.create_folder( "Support Knowledge Base", "Customer support documentation" ) # Upload support docs for doc in support_docs: self.hippo.upload_file(self.folder.id, doc) # Create chat self.chat = self.hippo.create_chat( self.folder.id, "Customer Support Chat" ) def ask(self, question): """Ask a support question""" answer = self.hippo.submit_ask(self.chat.id, question) # Format response response = { 'answer': answer.response, 'confidence': answer.confidence_score, 'sources': [ { 'file': s.file_name, 'page': s.page_number, 'relevance': s.relevance_score } for s in answer.sources ] } return response def get_history(self): """Get all Q&A history""" return self.hippo.get_asks(self.chat.id) # Usage bot = SupportBot(api_key="your-api-key") bot.setup(["faq.pdf", "user-guide.pdf", "troubleshooting.pdf"]) # Ask questions questions = [ "How do I reset my password?", "What payment methods are accepted?", "How long does shipping take?" ] for q in questions: result = bot.ask(q) print(f"\nQ: {q}") print(f"A: {result['answer']}") print(f"Confidence: {result['confidence']:.2f}") print(f"Sources: {len(result['sources'])}") ``` ## Best Practices ```python theme={null} answer = hippo.submit_ask(chat_id, question) if answer.confidence_score < 0.7: print("Low confidence - verifying sources...") for source in answer.sources: print(f"Check: {source.file_name}, page {source.page_number}") ``` Always check sources for low-confidence answers. ```python theme={null} # First attempt answer1 = hippo.submit_ask(chat_id, "What's the policy?") if answer1.confidence_score < 0.7: # Rephrase with more context answer2 = hippo.submit_ask( chat_id, "What is the company's refund policy for products?" ) ``` More specific questions → Better answers ```python theme={null} answer = hippo.submit_ask(chat_id, "What is the SLA?") if len(answer.sources) == 0: print("No sources found - may need to upload SLA document") # Upload missing documentation hippo.upload_file(folder_id, "service-level-agreement.pdf") # Ask again answer = hippo.submit_ask(chat_id, "What is the SLA?") ``` ```python theme={null} # Initial question a1 = hippo.submit_ask(chat_id, "How do I authenticate?") # If answer mentions multiple methods, follow up a2 = hippo.submit_ask(chat_id, "Which method is recommended?") # Further clarification a3 = hippo.submit_ask(chat_id, "Can you show an example?") ``` Conversation flow leads to better understanding ## Next Steps Optimize answer quality and costs Complete RAG workflow examples Organize your knowledge bases # Hippo Quickstart Source: https://docs.cerevox.ai/hippo/quickstart Build your first RAG Q&A system in 5 minutes # Hippo Quickstart - 5 Minutes to Q\&A 🦛 Build an AI Q\&A system that answers questions from your documents with source citations. ## Prerequisites **Before you start:** * Python 3.9+ installed * Cerevox API key ([get one here](https://cerevox.ai)) * `pip install cerevox` completed ## The 4-Step Workflow Hippo follows a simple pattern: ```mermaid theme={null} graph LR A[Create Folder] --> B[Upload Files] B --> C[Create Chat] C --> D[Ask Questions] style D fill:#0285c7 ``` ## Step-by-Step Implementation Folders organize documents into searchable knowledge bases. ```python theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key") # Create a folder for your documents folder = hippo.create_folder( name="Product Documentation", description="User guides and API docs" ) print(f"Created folder: {folder.name}") print(f"Folder ID: {folder.id}") ``` **Tip**: Use descriptive folder names - they help with organization when you have multiple knowledge bases. Add documents to your folder from files or URLs. ```python theme={null} # Upload from local file file1 = hippo.upload_file( folder_id=folder.id, file_path="user-guide.pdf" ) # Upload from URL file2 = hippo.upload_file_from_url( folder_id=folder.id, file_url="https://example.com/api-docs.pdf", file_name="api-docs.pdf" ) print(f"Uploaded: {file1.name}") print(f"Uploaded: {file2.name}") ``` **Supported formats**: PDF, DOCX, PPTX, XLSX, TXT, HTML, CSV, and more Large files (100+ pages) may take 2-5 minutes to process. Use async API for better performance. Chat sessions maintain conversation context for Q\&A. ```python theme={null} # Create chat linked to the folder chat = hippo.create_chat( folder_id=folder.id, chat_name="Technical Support Q&A" ) print(f"Created chat: {chat.name}") print(f"Chat ID: {chat.id}") ``` **Multiple chats per folder**: You can create different chats for different purposes (e.g., "Customer Support", "Internal Q\&A"). Submit questions and get AI-powered answers with citations! ```python theme={null} # Ask a question answer = hippo.submit_ask( chat_id=chat.id, question="How do I authenticate users in the API?" ) # Print the answer print(f"Question: {answer.question}") print(f"Answer: {answer.response}") print(f"Confidence: {answer.confidence_score}") # Show source citations print(f"\nSources ({len(answer.sources)} citations):") for source in answer.sources: print(f" - {source.file_name} (Page {source.page_number})") ``` **You're done!** You've built a RAG Q\&A system with 80% cost savings! 🎉 ## Complete Code Example ```python Full Workflow theme={null} from cerevox import Hippo # Initialize Hippo hippo = Hippo(api_key="your-api-key") # 1. Create folder folder = hippo.create_folder( name="Product Documentation", description="User guides and API docs" ) # 2. Upload files file1 = hippo.upload_file(folder.id, "user-guide.pdf") file2 = hippo.upload_file_from_url( folder.id, "https://example.com/api-docs.pdf", "api-docs.pdf" ) # 3. Create chat chat = hippo.create_chat(folder.id, "Technical Support") # 4. Ask questions questions = [ "How do I authenticate?", "What are the API rate limits?", "How do I handle errors?" ] for question in questions: answer = hippo.submit_ask(chat.id, question) print(f"\nQ: {question}") print(f"A: {answer.response}") print(f"Sources: {len(answer.sources)} citations") ``` ```python Async Version theme={null} import asyncio from cerevox import AsyncHippo async def main(): async with AsyncHippo(api_key="your-api-key") as hippo: # 1. Create folder folder = await hippo.create_folder("Product Docs") # 2. Upload files concurrently files = await asyncio.gather( hippo.upload_file(folder.id, "guide.pdf"), hippo.upload_file(folder.id, "docs.pdf") ) # 3. Create chat chat = await hippo.create_chat(folder.id, "Support") # 4. Ask questions answer = await hippo.submit_ask( chat.id, "How do I get started?" ) print(f"Answer: {answer.response}") print(f"Sources: {len(answer.sources)}") # Run async workflow asyncio.run(main()) ``` ## What You Get Back When you submit a question, Hippo returns: The AI-generated answer to your question The original question (as processed) Confidence score (0-1) indicating answer quality List of source documents with citations * `file_name`: Name of the source document * `file_id`: Unique file identifier * `page_number`: Page where information was found * `relevance_score`: How relevant this source is ## Testing Your Setup Run this verification script: ```python theme={null} from cerevox import Hippo def test_hippo(): hippo = Hippo(api_key="your-api-key") # Quick test folder = hippo.create_folder("Test Folder") # Upload test content test_file = hippo.upload_file(folder.id, "test.pdf") # Create chat and ask chat = hippo.create_chat(folder.id, "Test Chat") answer = hippo.submit_ask(chat.id, "What is this document about?") # Verify if answer and answer.response: print("✅ Hippo is working correctly!") print(f"Answer: {answer.response}") return True else: print("❌ Something went wrong") return False # Run test test_hippo() ``` ## Common First Questions * **Small files** (\< 10 pages): 10-30 seconds * **Medium files** (10-100 pages): 30-120 seconds * **Large files** (> 100 pages): 2-5 minutes Files are processed automatically in the background. You can ask questions as soon as upload completes - the system will wait for indexing to finish. Hippo pricing is based on: * Number of documents uploaded * Number of questions asked * Processing complexity **80% cheaper** than traditional RAG with full document retrieval! Check [pricing](https://cerevox.ai/pricing) for current rates. Supported formats: * **Documents**: PDF, DOCX, PPTX, TXT, RTF * **Spreadsheets**: XLSX, CSV * **Web**: HTML, MHTML * **Others**: Contact support for custom formats Max file size: 100MB per file (contact for larger files) Hippo supports **100+ languages** for both documents and questions, including: * English, Spanish, French, German, Italian * Chinese (Simplified & Traditional), Japanese, Korean * Arabic, Hebrew, Hindi, and more Same accuracy across all languages! Yes! Cerevox is enterprise-ready: * **Data encryption** at rest and in transit * **SOC 2 compliance** (in progress) * **Privacy controls**: Delete data anytime * **No training**: Your data is never used to train models See our [privacy policy](/legal/privacy) for details. ## Next Steps Learn to organize documents effectively Advanced file upload and management Manage conversations and context Optimize answer quality and costs *** **Need help?** Join our [Discord community](https://discord.gg/cerevox) or check out [complete examples](/examples/rag-workflow). # Privacy Policy Source: https://docs.cerevox.ai/legal/privacy Last updated: January 1, 2025 This Privacy Policy describes Our policies and procedures on the collection, use and disclosure of Your information when You use the Service and tells You about Your privacy rights and how the law protects You. We use Your Personal data to provide and improve the Service. By using the Service, You agree to the collection and use of information in accordance with this Privacy Policy. # Interpretation and Definitions ## Interpretation The words of which the initial letter is capitalized have meanings defined under the following conditions. The following definitions shall have the same meaning regardless of whether they appear in singular or in plural. ## Definitions For the purposes of this Privacy Policy: * **Account** means a unique account created for You to access our Service or parts of our Service. * **Affiliate** means an entity that controls, is controlled by or is under common control with a party, where "control" means ownership of 50% or more of the shares, equity interest or other securities entitled to vote for election of directors or other managing authority. * **Application** refers to Cerevox, the software program provided by the Company. * **Company** (referred to as either "the Company", "We", "Us" or "Our" in this Agreement) refers to Cerevox, Inc., 536 Middlebury Dr. Sunnyvale, CA 94087. * **Cookies** are small files that are placed on Your computer, mobile device or any other device by a website, containing the details of Your browsing history on that website among its many uses. * **Country** refers to: California, United States * **Device** means any device that can access the Service such as a computer, a cellphone or a digital tablet. * **Personal Data** is any information that relates to an identified or identifiable individual. * **Service** refers to the Application or the Website or both. * **Service Provider** means any natural or legal person who processes the data on behalf of the Company. It refers to third-party companies or individuals employed by the Company to facilitate the Service, to provide the Service on behalf of the Company, to perform services related to the Service or to assist the Company in analyzing how the Service is used. * **Third-party Social Media Service** refers to any website or any social network website through which a User can log in or create an account to use the Service. * **Usage Data** refers to data collected automatically, either generated by the use of the Service or from the Service infrastructure itself (for example, the duration of a page visit). * **Website** refers to Cerevox, accessible from [https://cerevox.ai](https://cerevox.ai) * **You** means the individual accessing or using the Service, or the company, or other legal entity on behalf of which such individual is accessing or using the Service, as applicable. # Collecting and Using Your Personal Data ## Types of Data Collected ### Personal Data While using Our Service, We may ask You to provide Us with certain personally identifiable information that can be used to contact or identify You. Personally identifiable information may include, but is not limited to: * Email address * First name and last name * Usage Data * Uploaded Data ### Usage Data Usage Data is collected automatically when using the Service. Usage Data may include information such as Your Device's Internet Protocol address (e.g. IP address), browser type, browser version, the pages of our Service that You visit, the time and date of Your visit, the time spent on those pages, unique device identifiers and other diagnostic data. When You access the Service by or through a mobile device, We may collect certain information automatically, including, but not limited to, the type of mobile device You use, Your mobile device unique ID, the IP address of Your mobile device, Your mobile operating system, the type of mobile Internet browser You use, unique device identifiers and other diagnostic data. We may also collect information that Your browser sends whenever You visit our Service or when You access the Service by or through a mobile device. ### Information from Third-Party Social Media Services The Company allows You to create an account and log in to use the Service through the following Third-party Social Media Services: * Google * Facebook * Instagram * Twitter * LinkedIn If You decide to register through or otherwise grant us access to a Third-Party Social Media Service, We may collect Personal data that is already associated with Your Third-Party Social Media Service's account, such as Your name, Your email address, Your activities or Your contact list associated with that account. You may also have the option of sharing additional information with the Company through Your Third-Party Social Media Service's account. If You choose to provide such information and Personal Data, during registration or otherwise, You are giving the Company permission to use, share, and store it in a manner consistent with this Privacy Policy. ## Google Drive Data Usage ### Collection and Storage of Google Drive Data We allow users to upload and integrate data from their Google Drive accounts into our Service. When you choose to connect your Google Drive account to our Service, we collect and store data from the files you select to upload. This data is stored securely in our systems to enable our core Service functionality, specifically to allow you to query your data for accurate sources and answers. ### Use of Google Drive Data Your Google Drive data is used exclusively to: * Provide you with query and search capabilities within your uploaded content * Generate accurate responses and sources based on your stored information * Maintain the functionality of our Service ### Data Access and Security Access to your Google Drive data is strictly limited to authorized personnel who require access to maintain and improve the Service We implement appropriate technical and organizational measures to protect your data We do not share your Google Drive data with third parties unless required by law or with your explicit consent ### User Control and Data Deletion You have complete control over your Google Drive data within our Service: * You can choose which files to upload and integrate * You can request the deletion of specific Google Drive data at any time * Upon account deletion, all associated Google Drive data is permanently removed from our systems * You can revoke our access to your Google Drive at any time through your Google Account settings ### Data Retention We retain your Google Drive data only for as long as necessary to provide our Service or until: * You specifically request the deletion of the data * You delete your account with our Service * You revoke our access to your Google Drive ### Employee Access Our employees may access your Google Drive data only when: * Specifically authorized for essential Service maintenance * Necessary to comply with legal obligations * Explicitly permitted by you All employee access is logged and monitored to ensure compliance with our security policies. ### Tracking Technologies and Cookies We use Cookies and similar tracking technologies to track the activity on Our Service and store certain information. Tracking technologies used are beacons, tags, and scripts to collect and track information and to improve and analyze Our Service. The technologies We use may include: * **Cookies or Browser Cookies.** A cookie is a small file placed on Your Device. You can instruct Your browser to refuse all Cookies or to indicate when a Cookie is being sent. However, if You do not accept Cookies, You may not be able to use some parts of our Service. Unless you have adjusted Your browser setting so that it will refuse Cookies, our Service may use Cookies. * **Web Beacons.** Certain sections of our Service and our emails may contain small electronic files known as web beacons (also referred to as clear gifs, pixel tags, and single-pixel gifs) that permit the Company, for example, to count users who have visited those pages or opened an email and for other related website statistics (for example, recording the popularity of a certain section and verifying system and server integrity). Cookies can be "Persistent" or "Session" Cookies. Persistent Cookies remain on Your personal computer or mobile device when You go offline, while Session Cookies are deleted as soon as You close Your web browser. You can learn more about cookies on [TermsFeed website](https://www.termsfeed.com/blog/cookies/#What_Are_Cookies) article. We use both Session and Persistent Cookies for the purposes set out below: * **Necessary / Essential Cookies** Type: Session Cookies Administered by: Us Purpose: These Cookies are essential to provide You with services available through the Website and to enable You to use some of its features. They help to authenticate users and prevent fraudulent use of user accounts. Without these Cookies, the services that You have asked for cannot be provided, and We only use these Cookies to provide You with those services. * **Cookies Policy / Notice Acceptance Cookies** Type: Persistent Cookies Administered by: Us Purpose: These Cookies identify if users have accepted the use of cookies on the Website. * **Functionality Cookies** Type: Persistent Cookies Administered by: Us Purpose: These Cookies allow us to remember choices You make when You use the Website, such as remembering your login details or language preference. The purpose of these Cookies is to provide You with a more personal experience and to avoid You having to re-enter your preferences every time You use the Website. For more information about the cookies we use and your choices regarding cookies, please visit our Cookies Policy or the Cookies section of our Privacy Policy. ## Use of Your Personal Data The Company may use Personal Data for the following purposes: * **To provide and maintain our Service**, including to monitor the usage of our Service. * **To manage Your Account:** to manage Your registration as a user of the Service. The Personal Data You provide can give You access to different functionalities of the Service that are available to You as a registered user. * **For the performance of a contract:** the development, compliance and undertaking of the purchase contract for the products, items or services You have purchased or of any other contract with Us through the Service. * **To contact You:** To contact You by email, telephone calls, SMS, or other equivalent forms of electronic communication, such as a mobile application's push notifications regarding updates or informative communications related to the functionalities, products or contracted services, including the security updates, when necessary or reasonable for their implementation. * **To provide You** with news, special offers and general information about other goods, services and events which we offer that are similar to those that you have already purchased or enquired about unless You have opted not to receive such information. * **To manage Your requests:** To attend and manage Your requests to Us. * **For business transfers:** We may use Your information to evaluate or conduct a merger, divestiture, restructuring, reorganization, dissolution, or other sale or transfer of some or all of Our assets, whether as a going concern or as part of bankruptcy, liquidation, or similar proceeding, in which Personal Data held by Us about our Service users is among the assets transferred. * **For other purposes**: We may use Your information for other purposes, such as data analysis, identifying usage trends, determining the effectiveness of our promotional campaigns and to evaluate and improve our Service, products, services, marketing and your experience. We may share Your personal information in the following situations: * **With Service Providers:** We may share Your personal information with Service Providers to monitor and analyze the use of our Service, to contact You. * **For business transfers:** We may share or transfer Your personal information in connection with, or during negotiations of, any merger, sale of Company assets, financing, or acquisition of all or a portion of Our business to another company. * **With Affiliates:** We may share Your information with Our affiliates, in which case we will require those affiliates to honor this Privacy Policy. Affiliates include Our parent company and any other subsidiaries, joint venture partners or other companies that We control or that are under common control with Us. * **With business partners:** We may share Your information with Our business partners to offer You certain products, services or promotions. * **With other users:** when You share personal information or otherwise interact in the public areas with other users, such information may be viewed by all users and may be publicly distributed outside. If You interact with other users or register through a Third-Party Social Media Service, Your contacts on the Third-Party Social Media Service may see Your name, profile, pictures and description of Your activity. Similarly, other users will be able to view descriptions of Your activity, communicate with You and view Your profile. * **With Your consent**: We may disclose Your personal information for any other purpose with Your consent. ## Retention of Your Personal Data The Company will retain Your Personal Data only for as long as is necessary for the purposes set out in this Privacy Policy. We will retain and use Your Personal Data to the extent necessary to comply with our legal obligations (for example, if we are required to retain your data to comply with applicable laws), resolve disputes, and enforce our legal agreements and policies. The Company will also retain Usage Data for internal analysis purposes. Usage Data is generally retained for a shorter period of time, except when this data is used to strengthen the security or to improve the functionality of Our Service, or We are legally obligated to retain this data for longer time periods. ## Transfer of Your Personal Data Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdiction where the data protection laws may differ than those from Your jurisdiction. Your consent to this Privacy Policy followed by Your submission of such information represents Your agreement to that transfer. The Company will take all steps reasonably necessary to ensure that Your data is treated securely and in accordance with this Privacy Policy and no transfer of Your Personal Data will take place to an organization or a country unless there are adequate controls in place including the security of Your data and other personal information. ## Delete Your Personal Data You have the right to delete or request that We assist in deleting the Personal Data that We have collected about You. Our Service may give You the ability to delete certain information about You from within the Service. You may update, amend, or delete Your information at any time by signing in to Your Account, if you have one, and visiting the account settings section that allows you to manage Your personal information. You may also contact Us to request access to, correct, or delete any personal information that You have provided to Us. Please note, however, that We may need to retain certain information when we have a legal obligation or lawful basis to do so. ## Disclosure of Your Personal Data ### Business Transactions If the Company is involved in a merger, acquisition or asset sale, Your Personal Data may be transferred. We will provide notice before Your Personal Data is transferred and becomes subject to a different Privacy Policy. ### Law enforcement Under certain circumstances, the Company may be required to disclose Your Personal Data if required to do so by law or in response to valid requests by public authorities (e.g. a court or a government agency). ### Other legal requirements The Company may disclose Your Personal Data in the good faith belief that such action is necessary to: * Comply with a legal obligation * Protect and defend the rights or property of the Company * Prevent or investigate possible wrongdoing in connection with the Service * Protect the personal safety of Users of the Service or the public * Protect against legal liability ## Security of Your Personal Data The security of Your Personal Data is important to Us, but remember that no method of transmission over the Internet, or method of electronic storage is 100% secure. While We strive to use commercially acceptable means to protect Your Personal Data, We cannot guarantee its absolute security. # Children's Privacy Our Service does not address anyone under the age of 13. We do not knowingly collect personally identifiable information from anyone under the age of 13. If You are a parent or guardian and You are aware that Your child has provided Us with Personal Data, please contact Us. If We become aware that We have collected Personal Data from anyone under the age of 13 without verification of parental consent, We take steps to remove that information from Our servers. If We need to rely on consent as a legal basis for processing Your information and Your country requires consent from a parent, We may require Your parent's consent before We collect and use that information. # Links to Other Websites Our Service may contain links to other websites that are not operated by Us. If You click on a third party link, You will be directed to that third party's site. We strongly advise You to review the Privacy Policy of every site You visit. We have no control over and assume no responsibility for the content, privacy policies or practices of any third party sites or services. # Changes to this Privacy Policy We may update Our Privacy Policy from time to time. We will notify You of any changes by posting the new Privacy Policy on this page. We will let You know via email and/or a prominent notice on Our Service, prior to the change becoming effective and update the "Last updated" date at the top of this Privacy Policy. You are advised to review this Privacy Policy periodically for any changes. Changes to this Privacy Policy are effective when they are posted on this page. # Contact Us If you have any questions about this Privacy Policy, You can contact us: * By email: [support@cerevox.ai](mailto:support@cerevox.ai) # Terms and Conditions Source: https://docs.cerevox.ai/legal/terms Last updated: January 1, 2025 Please read these terms and conditions carefully before using Our Service. # Interpretation and Definitions ## Interpretation The words of which the initial letter is capitalized have meanings defined under the following conditions. The following definitions shall have the same meaning regardless of whether they appear in singular or in plural. ## Definitions For the purposes of these Terms and Conditions: * **Application** means the software program provided by the Company downloaded by You on any electronic device, named Cerevox * **Application Store** means the digital distribution service operated and developed by Apple Inc. (Apple App Store) or Google Inc. (Google Play Store) in which the Application has been downloaded. * **Affiliate** means an entity that controls, is controlled by or is under common control with a party, where "control" means ownership of 50% or more of the shares, equity interest or other securities entitled to vote for election of directors or other managing authority. * **Account** means a unique account created for You to access our Service or parts of our Service. * **Country** refers to: California, United States * **Company** (referred to as either "the Company", "We", "Us" or "Our" in this Agreement) refers to Cerevox, Inc., 536 Middlebury Dr. Sunnyvale, CA 94087. * **Device** means any device that can access the Service such as a computer, a cellphone or a digital tablet. * **Free Trial** refers to a limited period of time that may be free when purchasing a Subscription. * **Service** refers to the Application or the Website or both. * **Subscriptions** refer to the services or access to the Service offered on a subscription basis by the Company to You. * **Terms and Conditions** (also referred as "Terms") mean these Terms and Conditions that form the entire agreement between You and the Company regarding the use of the Service. * **Third-party Social Media Service** means any services or content (including data, information, products or services) provided by a third-party that may be displayed, included or made available by the Service. * **Website** refers to Cerevox, accessible from [https://cerevox.ai](https://cerevox.ai) * **You** means the individual accessing or using the Service, or the company, or other legal entity on behalf of which such individual is accessing or using the Service, as applicable. # Acknowledgment These are the Terms and Conditions governing the use of this Service and the agreement that operates between You and the Company. These Terms and Conditions set out the rights and obligations of all users regarding the use of the Service. Your access to and use of the Service is conditioned on Your acceptance of and compliance with these Terms and Conditions. These Terms and Conditions apply to all visitors, users and others who access or use the Service. By accessing or using the Service You agree to be bound by these Terms and Conditions. If You disagree with any part of these Terms and Conditions then You may not access the Service. You represent that you are over the age of 18. The Company does not permit those under 18 to use the Service. Your access to and use of the Service is also conditioned on Your acceptance of and compliance with the Privacy Policy of the Company. Our Privacy Policy describes Our policies and procedures on the collection, use and disclosure of Your personal information when You use the Application or the Website and tells You about Your privacy rights and how the law protects You. Please read Our Privacy Policy carefully before using Our Service. # Subscriptions ## Subscription period The Service or some parts of the Service are available only with a paid Subscription. You will be billed in advance on a recurring and periodic basis (such as daily, weekly, monthly or annually), depending on the type of Subscription plan you select when purchasing the Subscription. At the end of each period, Your Subscription will automatically renew under the exact same conditions unless You cancel it or the Company cancels it. ## Subscription cancellations You may cancel Your Subscription renewal either through Your Account settings page or by contacting the Company. You will not receive a refund for the fees You already paid for Your current Subscription period and You will be able to access the Service until the end of Your current Subscription period. ## Billing You shall provide the Company with accurate and complete billing information including full name, address, state, zip code, telephone number, and a valid payment method information. Should automatic billing fail to occur for any reason, the Company will issue an electronic invoice indicating that you must proceed manually, within a certain deadline date, with the full payment corresponding to the billing period as indicated on the invoice. ## Fee Changes The Company, in its sole discretion and at any time, may modify the Subscription fees. Any Subscription fee change will become effective at the end of the then-current Subscription period. The Company will provide You with reasonable prior notice of any change in Subscription fees to give You an opportunity to terminate Your Subscription before such change becomes effective. Your continued use of the Service after the Subscription fee change comes into effect constitutes Your agreement to pay the modified Subscription fee amount. ## Refunds Except when required by law, paid Subscription fees are non-refundable. Certain refund requests for Subscriptions may be considered by the Company on a case-by-case basis and granted at the sole discretion of the Company. ## Free Trial The Company may, at its sole discretion, offer a Subscription with a Free Trial for a limited period of time. You may be required to enter Your billing information in order to sign up for the Free Trial. If You do enter Your billing information when signing up for a Free Trial, You will not be charged by the Company until the Free Trial has expired. On the last day of the Free Trial period, unless You canceled Your Subscription, You will be automatically charged the applicable Subscription fees for the type of Subscription You have selected. At any time and without notice, the Company reserves the right to (i) modify the terms and conditions of the Free Trial offer, or (ii) cancel such Free Trial offer. # User Accounts When You create an account with Us, You must provide Us information that is accurate, complete, and current at all times. Failure to do so constitutes a breach of the Terms, which may result in immediate termination of Your account on Our Service. You are responsible for safeguarding the password that You use to access the Service and for any activities or actions under Your password, whether Your password is with Our Service or a Third-Party Social Media Service. You agree not to disclose Your password to any third party. You must notify Us immediately upon becoming aware of any breach of security or unauthorized use of Your account. You may not use as a username the name of another person or entity or that is not lawfully available for use, a name or trademark that is subject to any rights of another person or entity other than You without appropriate authorization, or a name that is otherwise offensive, vulgar or obscene. # Intellectual Property The Service and its original content (excluding Content provided by You or other users), features and functionality are and will remain the exclusive property of the Company and its licensors. The Service is protected by copyright, trademark, and other laws of both the Country and foreign countries. Our trademarks and trade dress may not be used in connection with any product or service without the prior written consent of the Company. # Links to Other Websites Our Service may contain links to third-party web sites or services that are not owned or controlled by the Company. The Company has no control over, and assumes no responsibility for, the content, privacy policies, or practices of any third party web sites or services. You further acknowledge and agree that the Company shall not be responsible or liable, directly or indirectly, for any damage or loss caused or alleged to be caused by or in connection with the use of or reliance on any such content, goods or services available on or through any such web sites or services. We strongly advise You to read the terms and conditions and privacy policies of any third-party web sites or services that You visit. # Termination We may terminate or suspend Your Account immediately, without prior notice or liability, for any reason whatsoever, including without limitation if You breach these Terms and Conditions. Upon termination, Your right to use the Service will cease immediately. If You wish to terminate Your Account, You may simply discontinue using the Service. # Limitation of Liability Notwithstanding any damages that You might incur, the entire liability of the Company and any of its suppliers under any provision of this Terms and Your exclusive remedy for all of the foregoing shall be limited to the amount actually paid by You through the Service or 100 USD if You haven't purchased anything through the Service. To the maximum extent permitted by applicable law, in no event shall the Company or its suppliers be liable for any special, incidental, indirect, or consequential damages whatsoever (including, but not limited to, damages for loss of profits, loss of data or other information, for business interruption, for personal injury, loss of privacy arising out of or in any way related to the use of or inability to use the Service, third-party software and/or third-party hardware used with the Service, or otherwise in connection with any provision of this Terms), even if the Company or any supplier has been advised of the possibility of such damages and even if the remedy fails of its essential purpose. Some states do not allow the exclusion of implied warranties or limitation of liability for incidental or consequential damages, which means that some of the above limitations may not apply. In these states, each party's liability will be limited to the greatest extent permitted by law. # "AS IS" and "AS AVAILABLE" Disclaimer The Service is provided to You "AS IS" and "AS AVAILABLE" and with all faults and defects without warranty of any kind. To the maximum extent permitted under applicable law, the Company, on its own behalf and on behalf of its Affiliates and its and their respective licensors and service providers, expressly disclaims all warranties, whether express, implied, statutory or otherwise, with respect to the Service, including all implied warranties of merchantability, fitness for a particular purpose, title and non-infringement, and warranties that may arise out of course of dealing, course of performance, usage or trade practice. Without limitation to the foregoing, the Company provides no warranty or undertaking, and makes no representation of any kind that the Service will meet Your requirements, achieve any intended results, be compatible or work with any other software, applications, systems or services, operate without interruption, meet any performance or reliability standards or be error free or that any errors or defects can or will be corrected. Without limiting the foregoing, neither the Company nor any of the company's provider makes any representation or warranty of any kind, express or implied: (i) as to the operation or availability of the Service, or the information, content, and materials or products included thereon; (ii) that the Service will be uninterrupted or error-free; (iii) as to the accuracy, reliability, or currency of any information or content provided through the Service; or (iv) that the Service, its servers, the content, or e-mails sent from or on behalf of the Company are free of viruses, scripts, trojan horses, worms, malware, timebombs or other harmful components. Some jurisdictions do not allow the exclusion of certain types of warranties or limitations on applicable statutory rights of a consumer, so some or all of the above exclusions and limitations may not apply to You. But in such a case the exclusions and limitations set forth in this section shall be applied to the greatest extent enforceable under applicable law. # Governing Law The laws of the Country, excluding its conflicts of law rules, shall govern this Terms and Your use of the Service. Your use of the Application may also be subject to other local, state, national, or international laws. # Disputes Resolution If You have any concern or dispute about the Service, You agree to first try to resolve the dispute informally by contacting the Company. # For European Union (EU) Users If You are a European Union consumer, you will benefit from any mandatory provisions of the law of the country in which you are resident in. # United States Federal Government End Use Provisions If You are a U.S. federal government end user, our Service is a "Commercial Item" as that term is defined at 48 C.F.R. §2.101. # United States Legal Compliance You represent and warrant that (i) You are not located in a country that is subject to the United States government embargo, or that has been designated by the United States government as a "terrorist supporting" country, and (ii) You are not listed on any United States government list of prohibited or restricted parties. # Severability and Waiver ## Severability If any provision of these Terms is held to be unenforceable or invalid, such provision will be changed and interpreted to accomplish the objectives of such provision to the greatest extent possible under applicable law and the remaining provisions will continue in full force and effect. ## Waiver Except as provided herein, the failure to exercise a right or to require performance of an obligation under these Terms shall not effect a party's ability to exercise such right or require such performance at any time thereafter nor shall the waiver of a breach constitute a waiver of any subsequent breach. # Translation Interpretation These Terms and Conditions may have been translated if We have made them available to You on our Service. You agree that the original English text shall prevail in the case of a dispute. # Changes to These Terms and Conditions We reserve the right, at Our sole discretion, to modify or replace these Terms at any time. If a revision is material We will make reasonable efforts to provide at least 30 days' notice prior to any new terms taking effect. What constitutes a material change will be determined at Our sole discretion. By continuing to access or use Our Service after those revisions become effective, You agree to be bound by the revised terms. If You do not agree to the new terms, in whole or in part, please stop using the website and the Service. # Contact Us If you have any questions about these Terms and Conditions, You can contact us: * By email: [support@cerevox.ai](mailto:support@cerevox.ai) # Financial Analysis Source: https://docs.cerevox.ai/usecases/financial Extract structured insights from financial documents with enterprise-grade accuracy # Financial Document Analysis with Lexa Transform complex financial documents into structured, analyzable data in seconds. Lexa's AI-powered parsing extracts critical financial metrics, tables, and insights with highest accuracy. ## Why Lexa for Financial Analysis? Preserve complex financial tables and calculations SOC 2 compliant with enterprise security Process 10-K filings, earnings reports, and prospectuses Extract insights from 100+ page reports in under 30 seconds ## Supported Financial Documents * **SEC Filings** (10-K, 10-Q, 8-K, proxy statements) * **Earnings Reports** and quarterly statements * **Annual Reports** with complex layouts * **Financial Statements** (income, balance sheet, cash flow) * **Research Reports** from analysts * **Prospectuses** and offering memoranda * **Credit Reports** and risk assessments ## Quick Start: Parse SEC 10-K Filing Transform a complex SEC filing into structured data: ```python Sync Example theme={null} from cerevox import Lexa # Initialize client client = Lexa(api_key="your-api-key") # Parse SEC 10-K filing documents = client.parse("tesla_10k.pdf") doc = documents[0] # Extract financial metrics revenue_chunks = doc.search_content("revenue|total revenue") profit_chunks = doc.search_content("net income|profit") print(f"Document: {doc.filename}") print(f"Pages: {doc.total_pages}") print(f"Tables found: {len(doc.tables)}") print(f"Financial metrics extracted: {len(revenue_chunks + profit_chunks)}") ``` ```python Async Example (Recommended) theme={null} import asyncio from cerevox import AsyncLexa async def analyze_financial_documents(): async with AsyncLexa(api_key="your-api-key") as client: # Process multiple financial documents files = ["10k_report.pdf", "earnings_q3.pdf", "balance_sheet.xlsx"] documents = await client.parse(files) financial_insights = [] for doc in documents: # Extract structured financial data insights = { 'filename': doc.filename, 'tables': len(doc.tables), 'revenue_mentions': len(doc.search_content("revenue")), 'risk_factors': len(doc.search_content("risk factor|risks")), 'chunks': doc.get_text_chunks(target_size=512) } financial_insights.append(insights) return financial_insights # Run analysis results = asyncio.run(analyze_financial_documents()) ``` ## Advanced Financial Analysis Patterns ### Extract Financial Tables Process complex financial statements and preserve structure: ```python theme={null} from cerevox import AsyncLexa import pandas as pd async def extract_financial_tables(filing_path): async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(filing_path) doc = documents[0] # Convert tables to pandas DataFrames financial_tables = [] for table in doc.tables: try: # Convert table to structured data df = pd.DataFrame(table.to_dict()) # Identify table type based on content table_type = identify_table_type(df) financial_tables.append({ 'type': table_type, 'data': df, 'page': table.page_number, 'rows': len(df), 'columns': len(df.columns) }) except Exception as e: print(f"Table processing error: {e}") continue return financial_tables def identify_table_type(df): """Identify financial table type based on content""" columns_text = ' '.join(df.columns.astype(str)).lower() if 'revenue' in columns_text or 'income' in columns_text: return 'income_statement' elif 'assets' in columns_text or 'liabilities' in columns_text: return 'balance_sheet' elif 'cash flow' in columns_text or 'operating activities' in columns_text: return 'cash_flow' else: return 'other_financial' # Usage tables = await extract_financial_tables("company_10k.pdf") for table in tables: print(f"Found {table['type']} with {table['rows']} rows") ``` ### Financial Metrics Extraction Build a comprehensive financial metrics extractor: ```python theme={null} import re from typing import Dict, List class FinancialMetricsExtractor: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) # Define financial metric patterns self.metric_patterns = { 'revenue': [ r'total revenue[:\s]+\$?([\d,\.]+)', r'net sales[:\s]+\$?([\d,\.]+)', r'revenue[:\s]+\$?([\d,\.]+)\s*(million|billion)?' ], 'profit': [ r'net income[:\s]+\$?([\d,\.]+)', r'profit[:\s]+\$?([\d,\.]+)', r'earnings[:\s]+\$?([\d,\.]+)' ], 'assets': [ r'total assets[:\s]+\$?([\d,\.]+)', r'assets[:\s]+\$?([\d,\.]+)' ], 'debt': [ r'total debt[:\s]+\$?([\d,\.]+)', r'long.term debt[:\s]+\$?([\d,\.]+)' ] } async def extract_metrics(self, document_path: str) -> Dict: """Extract financial metrics from document""" async with self.client: documents = await self.client.parse(document_path) doc = documents[0] metrics = {} content = doc.content.lower() for metric_type, patterns in self.metric_patterns.items(): values = [] for pattern in patterns: matches = re.findall(pattern, content, re.IGNORECASE) values.extend(matches) # Clean and convert values cleaned_values = [] for value in values: if isinstance(value, tuple): value = value[0] # Extract number from regex group # Remove commas and convert to float try: clean_value = float(value.replace(',', '')) cleaned_values.append(clean_value) except ValueError: continue metrics[metric_type] = cleaned_values return { 'document': document_path, 'metrics': metrics, 'summary': self._summarize_metrics(metrics) } def _summarize_metrics(self, metrics: Dict) -> Dict: """Generate summary statistics""" summary = {} for metric_type, values in metrics.items(): if values: summary[metric_type] = { 'count': len(values), 'max': max(values), 'min': min(values), 'avg': sum(values) / len(values) } return summary # Usage extractor = FinancialMetricsExtractor("your-api-key") results = await extractor.extract_metrics("annual_report.pdf") print(f"Revenue mentions: {len(results['metrics']['revenue'])}") print(f"Profit data points: {len(results['metrics']['profit'])}") ``` ### RAG for Financial Q\&A Build a financial document Q\&A system: ```python theme={null} from cerevox import AsyncLexa import openai from typing import List class FinancialRAGSystem: def __init__(self, cerevox_api_key: str, openai_api_key: str): self.cerevox_client = AsyncLexa(api_key=cerevox_api_key) openai.api_key = openai_api_key self.document_chunks = [] async def ingest_documents(self, financial_docs: List[str]): """Ingest and chunk financial documents""" async with self.cerevox_client: documents = await self.cerevox_client.parse(financial_docs) # Create vector-ready chunks all_chunks = documents.get_all_text_chunks( target_size=1000, # Optimal for financial context tolerance=0.15 ) # Add metadata for better retrieval for i, chunk in enumerate(all_chunks): chunk_with_metadata = { 'id': f'chunk_{i}', 'content': chunk, 'document': documents[i // len(all_chunks) * len(documents)].filename, 'embedding': await self._get_embedding(chunk) } self.document_chunks.append(chunk_with_metadata) print(f"Ingested {len(self.document_chunks)} chunks from financial documents") async def _get_embedding(self, text: str): """Get embedding for text chunk""" response = await openai.Embedding.acreate( model="text-embedding-ada-002", input=text ) return response['data'][0]['embedding'] async def query_financials(self, question: str, top_k: int = 5): """Answer questions about financial documents""" # Get question embedding question_embedding = await self._get_embedding(question) # Find relevant chunks (simplified - use proper vector DB in production) relevant_chunks = self._find_similar_chunks( question_embedding, top_k ) # Build context for LLM context = "\n\n".join([ f"Document: {chunk['document']}\nContent: {chunk['content']}" for chunk in relevant_chunks ]) # Generate answer response = await openai.ChatCompletion.acreate( model="gpt-4", messages=[ { "role": "system", "content": "You are a financial analyst assistant. Answer questions based on the provided financial document context. Be precise and cite specific numbers when available." }, { "role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}" } ] ) return { 'answer': response.choices[0].message.content, 'sources': [chunk['document'] for chunk in relevant_chunks], 'relevant_chunks': len(relevant_chunks) } def _find_similar_chunks(self, query_embedding, top_k): """Find most similar chunks (simplified implementation)""" # In production, use a proper vector database like Pinecone or Weaviate import numpy as np similarities = [] for chunk in self.document_chunks: similarity = np.dot(query_embedding, chunk['embedding']) similarities.append((similarity, chunk)) # Sort by similarity and return top k similarities.sort(key=lambda x: x[0], reverse=True) return [chunk for _, chunk in similarities[:top_k]] # Usage rag_system = FinancialRAGSystem( cerevox_api_key="your-cerevox-key", openai_api_key="your-openai-key" ) # Ingest financial documents await rag_system.ingest_documents([ "tesla_10k_2023.pdf", "tesla_q3_earnings.pdf", "tesla_annual_report.pdf" ]) # Ask questions result = await rag_system.query_financials( "What was Tesla's revenue growth in 2023?" ) print(f"Answer: {result['answer']}") print(f"Sources: {result['sources']}") ``` ## Real-World Financial Use Cases ### 1. Investment Research Automation ```python theme={null} async def automated_investment_research(company_docs): """Automate investment research from company filings""" async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(company_docs) research_data = {} for doc in documents: # Extract key investment metrics research_data[doc.filename] = { 'revenue_growth': extract_growth_metrics(doc, 'revenue'), 'profit_margins': extract_margins(doc), 'risk_factors': doc.search_content('risk factor|material risk'), 'management_discussion': extract_md_a(doc), 'financial_highlights': extract_highlights(doc) } return research_data def extract_growth_metrics(doc, metric): """Extract growth metrics from financial documents""" # Search for year-over-year comparisons growth_patterns = [ f'{metric}.*increased.*(\d+\.?\d*)%', f'{metric}.*growth.*(\d+\.?\d*)%', f'{metric}.*up.*(\d+\.?\d*)%' ] growth_data = [] for pattern in growth_patterns: matches = re.findall(pattern, doc.content, re.IGNORECASE) growth_data.extend(matches) return growth_data ``` ### 2. Risk Assessment Pipeline ```python theme={null} class FinancialRiskAnalyzer: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) async def analyze_risk_factors(self, financial_docs: List[str]): """Comprehensive risk factor analysis""" async with self.client: documents = await self.client.parse(financial_docs) risk_analysis = {} for doc in documents: # Extract risk sections risk_chunks = doc.search_content("risk factor|risks|uncertainties") # Categorize risks categorized_risks = self._categorize_risks(risk_chunks) # Calculate risk scores risk_scores = self._calculate_risk_scores(categorized_risks) risk_analysis[doc.filename] = { 'total_risk_mentions': len(risk_chunks), 'risk_categories': categorized_risks, 'risk_scores': risk_scores, 'high_priority_risks': self._identify_high_priority(risk_chunks) } return risk_analysis def _categorize_risks(self, risk_chunks): """Categorize financial risks""" categories = { 'market_risk': ['market', 'competition', 'demand'], 'operational_risk': ['operations', 'supply chain', 'manufacturing'], 'financial_risk': ['liquidity', 'credit', 'debt', 'cash flow'], 'regulatory_risk': ['regulation', 'compliance', 'legal'], 'technology_risk': ['cyber', 'technology', 'data breach'] } categorized = {cat: [] for cat in categories} for chunk in risk_chunks: chunk_lower = chunk.lower() for category, keywords in categories.items(): if any(keyword in chunk_lower for keyword in keywords): categorized[category].append(chunk) return categorized ``` ### 3. Earnings Call Analysis ```python theme={null} async def analyze_earnings_transcripts(transcript_files): """Analyze earnings call transcripts for sentiment and insights""" async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(transcript_files) earnings_insights = {} for doc in documents: # Extract Q&A sections qa_sections = doc.search_content("questions and answers|q&a") # Management guidance guidance = doc.search_content("guidance|outlook|forecast") # Key metrics mentioned metrics = extract_financial_metrics(doc.content) earnings_insights[doc.filename] = { 'qa_insights': len(qa_sections), 'forward_guidance': guidance, 'key_metrics': metrics, 'sentiment_indicators': analyze_sentiment(doc.content) } return earnings_insights ``` ## Performance Benchmarks Lexa delivers exceptional performance for financial document processing: **30 seconds** average for 100+ page 10-K filing **99.8%** accuracy on complex financial tables **500+ documents/hour** with async processing ## Integration Examples ### Pinecone Vector Database ```python theme={null} import pinecone from cerevox import AsyncLexa # Initialize Pinecone pinecone.init(api_key="your-pinecone-key", environment="your-env") index = pinecone.Index("financial-docs") async def index_financial_documents(document_paths): async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(document_paths) # Get optimized chunks for financial documents chunks = documents.get_all_text_chunks( target_size=1000, # Good for financial context tolerance=0.1 ) # Create embeddings and upsert to Pinecone vectors = [] for i, chunk in enumerate(chunks): vector = { 'id': f'financial_chunk_{i}', 'values': get_embedding(chunk), 'metadata': { 'content': chunk, 'document_type': 'financial', 'source': documents[i // len(chunks) * len(documents)].filename } } vectors.append(vector) # Batch upsert to Pinecone index.upsert(vectors=vectors) return f"Indexed {len(vectors)} financial document chunks" ``` ## Security & Compliance Lexa is **SOC 2 Type II certified** and provides enterprise-grade security for sensitive financial documents. * **Data Encryption**: End-to-end encryption in transit and at rest * **Access Controls**: Role-based access with audit logging * **Compliance**: SOC 2, GDPR, and financial industry standards * **Data Residency**: Control where your financial data is processed ## Next Steps Ready to transform your financial document analysis? Process your first financial document in 3 minutes Explore advanced parsing methods Build financial RAG applications Optimize for large financial datasets # Knowledge Management Source: https://docs.cerevox.ai/usecases/knowledge Build intelligent knowledge bases and RAG systems with enterprise-grade document processing # Knowledge Management with Lexa Transform your organization's documents into intelligent, searchable knowledge bases. Lexa's AI-powered parsing creates the foundation for next-generation knowledge management and RAG applications. ## Why Lexa for Knowledge Management? Vector-optimized chunks preserve context and meaning Handle 12+ file formats in a single workflow Process thousands of documents with async operations Maintain document structure and relationships ## Knowledge Base Applications * **Internal Documentation** (policies, procedures, handbooks) * **Training Materials** (onboarding docs, certification guides) * **Technical Documentation** (API docs, system manuals) * **Research Archives** (reports, whitepapers, studies) * **Customer Support** (FAQs, troubleshooting guides) * **Compliance Documentation** (regulations, audit materials) * **Product Documentation** (user guides, specifications) ## Quick Start: Build a Knowledge Base Transform your document library into an intelligent knowledge system: ```python Basic Knowledge Base theme={null} from cerevox import Lexa # Initialize client client = Lexa(api_key="your-api-key") # Process knowledge base documents documents = client.parse([ "employee_handbook.pdf", "company_policies.docx", "technical_procedures.pdf" ]) # Create searchable knowledge chunks knowledge_chunks = [] for doc in documents: chunks = doc.get_text_chunks(target_size=512) for chunk in chunks: knowledge_chunks.append({ 'content': chunk, 'source': doc.filename, 'document_type': classify_document(doc), 'chunk_id': f"{doc.filename}_{chunks.index(chunk)}" }) print(f"Knowledge base: {len(knowledge_chunks)} searchable chunks") def classify_document(doc): """Classify document by content""" filename = doc.filename.lower() if 'policy' in filename: return 'policy' elif 'handbook' in filename: return 'handbook' elif 'procedure' in filename: return 'procedure' else: return 'general' ``` ```python Advanced RAG System theme={null} import asyncio from cerevox import AsyncLexa import openai from typing import List, Dict class KnowledgeRAGSystem: def __init__(self, cerevox_key: str, openai_key: str): self.cerevox_client = AsyncLexa(api_key=cerevox_key) openai.api_key = openai_key self.knowledge_base = [] async def build_knowledge_base(self, document_paths: List[str]): """Build searchable knowledge base""" async with self.cerevox_client: documents = await self.cerevox_client.parse(document_paths) # Create optimized chunks for knowledge retrieval all_chunks = documents.get_all_text_chunks( target_size=750, # Good for knowledge context tolerance=0.15 ) # Add metadata and embeddings for i, chunk in enumerate(all_chunks): doc_index = i // len(all_chunks) * len(documents) source_doc = documents[doc_index] entry = { 'id': f'kb_{i}', 'content': chunk, 'source': source_doc.filename, 'category': self.categorize_content(chunk), 'embedding': await self.get_embedding(chunk) } self.knowledge_base.append(entry) return f"Built knowledge base with {len(self.knowledge_base)} entries" async def query_knowledge(self, question: str, top_k: int = 3): """Query the knowledge base""" # Get question embedding question_emb = await self.get_embedding(question) # Find relevant chunks relevant = self.find_relevant_chunks(question_emb, top_k) # Generate answer with context context = "\n\n".join([chunk['content'] for chunk in relevant]) response = await openai.ChatCompletion.acreate( model="gpt-4", messages=[ { "role": "system", "content": "Answer questions using the provided knowledge base context. Be accurate and cite sources." }, { "role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}" } ] ) return { 'answer': response.choices[0].message.content, 'sources': [chunk['source'] for chunk in relevant], 'categories': list(set(chunk['category'] for chunk in relevant)) } def categorize_content(self, content: str) -> str: """Categorize content by type""" content_lower = content.lower() if any(term in content_lower for term in ['policy', 'rule', 'guideline']): return 'policy' elif any(term in content_lower for term in ['process', 'procedure', 'step']): return 'procedure' elif any(term in content_lower for term in ['benefit', 'compensation', 'leave']): return 'hr' elif any(term in content_lower for term in ['security', 'access', 'login']): return 'security' else: return 'general' # Usage rag_system = KnowledgeRAGSystem( cerevox_key="your-cerevox-key", openai_key="your-openai-key" ) # Build knowledge base await rag_system.build_knowledge_base([ "employee_handbook.pdf", "it_policies.pdf", "security_procedures.docx" ]) # Query knowledge result = await rag_system.query_knowledge("What is the remote work policy?") print(f"Answer: {result['answer']}") print(f"Sources: {result['sources']}") ``` ## Advanced Knowledge Management ### Multi-Source Knowledge Integration Combine documents from various sources into a unified knowledge base: ```python theme={null} from cerevox import AsyncLexa from typing import Dict, List import asyncio class EnterpriseKnowledgeManager: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) self.knowledge_domains = {} async def ingest_by_domain(self, domain_documents: Dict[str, List[str]]): """Ingest documents organized by knowledge domain""" async with self.client: for domain, document_paths in domain_documents.items(): print(f"Processing {domain} domain...") documents = await self.client.parse(document_paths) # Create domain-specific chunks domain_chunks = documents.get_all_text_chunks( target_size=600, tolerance=0.2 ) # Enhance with domain metadata processed_chunks = [] for i, chunk in enumerate(domain_chunks): doc_idx = i // len(domain_chunks) * len(documents) source_doc = documents[doc_idx] processed_chunks.append({ 'content': chunk, 'domain': domain, 'source': source_doc.filename, 'confidence': self.calculate_relevance(chunk, domain), 'keywords': self.extract_keywords(chunk), 'section_type': self.identify_section_type(chunk) }) self.knowledge_domains[domain] = processed_chunks print(f" Added {len(processed_chunks)} chunks to {domain}") total_chunks = sum(len(chunks) for chunks in self.knowledge_domains.values()) return f"Enterprise knowledge base: {total_chunks} chunks across {len(self.knowledge_domains)} domains" def search_domain(self, domain: str, query: str, limit: int = 5): """Search within a specific knowledge domain""" if domain not in self.knowledge_domains: return [] domain_chunks = self.knowledge_domains[domain] query_terms = query.lower().split() results = [] for chunk in domain_chunks: # Score based on term frequency and confidence term_score = sum(1 for term in query_terms if term in chunk['content'].lower()) total_score = term_score * chunk['confidence'] if total_score > 0: results.append({ 'content': chunk['content'], 'source': chunk['source'], 'domain': chunk['domain'], 'score': total_score, 'section_type': chunk['section_type'] }) # Sort by relevance results.sort(key=lambda x: x['score'], reverse=True) return results[:limit] def cross_domain_search(self, query: str, limit: int = 10): """Search across all knowledge domains""" all_results = [] for domain in self.knowledge_domains: domain_results = self.search_domain(domain, query, limit) all_results.extend(domain_results) # Sort all results by score all_results.sort(key=lambda x: x['score'], reverse=True) return all_results[:limit] def calculate_relevance(self, chunk: str, domain: str) -> float: """Calculate content relevance to domain""" # Simplified relevance scoring domain_keywords = { 'hr': ['employee', 'benefit', 'policy', 'leave', 'compensation'], 'it': ['system', 'security', 'access', 'software', 'network'], 'finance': ['budget', 'expense', 'accounting', 'revenue', 'cost'], 'legal': ['contract', 'compliance', 'regulation', 'liability'], 'operations': ['process', 'procedure', 'workflow', 'standard'] } if domain not in domain_keywords: return 0.5 # Neutral relevance keywords = domain_keywords[domain] chunk_lower = chunk.lower() matches = sum(1 for keyword in keywords if keyword in chunk_lower) return min(1.0, matches / len(keywords) + 0.3) def extract_keywords(self, text: str) -> List[str]: """Extract key terms from text""" import re from collections import Counter # Simple keyword extraction words = re.findall(r'\b[a-zA-Z]{3,}\b', text.lower()) # Filter common words stop_words = {'the', 'and', 'for', 'are', 'but', 'not', 'you', 'all', 'can', 'had', 'her', 'was', 'one', 'our', 'out', 'day', 'get', 'has', 'him', 'his', 'how', 'man', 'new', 'now', 'old', 'see', 'two', 'way', 'who', 'boy', 'did', 'its', 'let', 'put', 'say', 'she', 'too', 'use'} filtered_words = [word for word in words if word not in stop_words and len(word) > 3] # Return top keywords word_freq = Counter(filtered_words) return [word for word, _ in word_freq.most_common(5)] def identify_section_type(self, text: str) -> str: """Identify the type of content section""" text_lower = text.lower() if any(term in text_lower for term in ['procedure', 'step', 'process']): return 'procedure' elif any(term in text_lower for term in ['policy', 'rule', 'guideline']): return 'policy' elif any(term in text_lower for term in ['example', 'case study', 'scenario']): return 'example' elif any(term in text_lower for term in ['requirement', 'must', 'shall']): return 'requirement' else: return 'information' # Usage kb_manager = EnterpriseKnowledgeManager("your-api-key") # Organize documents by domain domain_docs = { 'hr': ['employee_handbook.pdf', 'benefits_guide.pdf'], 'it': ['security_policy.pdf', 'system_procedures.docx'], 'finance': ['expense_policy.pdf', 'budget_guidelines.pdf'], 'legal': ['compliance_guide.pdf', 'contract_templates.pdf'] } # Build domain-specific knowledge base await kb_manager.ingest_by_domain(domain_docs) # Search within specific domain hr_results = kb_manager.search_domain('hr', 'vacation policy') # Cross-domain search all_results = kb_manager.cross_domain_search('security requirements') ``` ### Knowledge Base Analytics Monitor and analyze your knowledge base performance: ```python theme={null} class KnowledgeAnalytics: def __init__(self, knowledge_base: List[Dict]): self.kb = knowledge_base def analyze_coverage(self): """Analyze knowledge base coverage""" from collections import Counter # Domain distribution domains = Counter(chunk['domain'] for chunk in self.kb) # Source document distribution sources = Counter(chunk['source'] for chunk in self.kb) # Content type distribution section_types = Counter(chunk['section_type'] for chunk in self.kb) return { 'total_chunks': len(self.kb), 'domains': dict(domains), 'sources': dict(sources), 'section_types': dict(section_types), 'avg_chunk_length': sum(len(chunk['content']) for chunk in self.kb) / len(self.kb) } def identify_gaps(self, query_log: List[str]): """Identify knowledge gaps from query patterns""" gap_analysis = {} for query in query_log: # Simplified gap detection query_terms = set(query.lower().split()) # Find chunks that might answer this query relevant_chunks = [] for chunk in self.kb: chunk_terms = set(chunk['content'].lower().split()) overlap = len(query_terms & chunk_terms) if overlap > 0: relevant_chunks.append({ 'chunk': chunk, 'overlap': overlap }) if not relevant_chunks: gap_analysis[query] = 'no_relevant_content' elif max(c['overlap'] for c in relevant_chunks) < 2: gap_analysis[query] = 'insufficient_coverage' return gap_analysis def suggest_improvements(self): """Suggest knowledge base improvements""" analysis = self.analyze_coverage() suggestions = [] # Check domain balance domain_counts = analysis['domains'] max_domain = max(domain_counts.values()) min_domain = min(domain_counts.values()) if max_domain > min_domain * 3: suggestions.append({ 'type': 'domain_imbalance', 'message': f'Consider adding more content to underrepresented domains', 'details': domain_counts }) # Check chunk size variation avg_length = analysis['avg_chunk_length'] if avg_length < 300: suggestions.append({ 'type': 'chunk_size', 'message': 'Chunks may be too small for good context', 'recommendation': 'Consider increasing target_size to 500-800' }) elif avg_length > 1200: suggestions.append({ 'type': 'chunk_size', 'message': 'Chunks may be too large for precise retrieval', 'recommendation': 'Consider decreasing target_size to 600-900' }) return suggestions # Usage with previous knowledge manager analytics = KnowledgeAnalytics( [chunk for chunks in kb_manager.knowledge_domains.values() for chunk in chunks] ) coverage = analytics.analyze_coverage() print(f"Knowledge base coverage: {coverage}") # Analyze query gaps sample_queries = [ "remote work policy", "expense reimbursement process", "security incident reporting", "performance review cycle" ] gaps = analytics.identify_gaps(sample_queries) suggestions = analytics.suggest_improvements() ``` ## Real-World Knowledge Management Use Cases ### Customer Support Knowledge Base ```python theme={null} async def build_support_knowledge_base(support_docs): """Build customer support knowledge base""" async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(support_docs) support_kb = [] for doc in documents: # Focus on Q&A and troubleshooting content chunks = doc.get_text_chunks(target_size=400) # Good for FAQ format for chunk in chunks: # Classify support content type content_type = classify_support_content(chunk) if content_type != 'irrelevant': support_kb.append({ 'content': chunk, 'type': content_type, 'source': doc.filename, 'priority': calculate_support_priority(chunk) }) return support_kb def classify_support_content(text): """Classify customer support content""" text_lower = text.lower() if any(term in text_lower for term in ['question', 'q:', 'faq', 'how to']): return 'faq' elif any(term in text_lower for term in ['error', 'issue', 'problem', 'troubleshoot']): return 'troubleshooting' elif any(term in text_lower for term in ['step', 'guide', 'instruction']): return 'guide' else: return 'general' def calculate_support_priority(text): """Calculate priority based on urgency indicators""" urgency_terms = ['critical', 'urgent', 'emergency', 'down', 'broken'] priority_score = sum(1 for term in urgency_terms if term in text.lower()) return min(5, priority_score + 1) # Scale 1-5 ``` ### Training Documentation System ```python theme={null} async def create_training_system(training_materials): """Create structured training documentation system""" async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(training_materials) training_modules = {} for doc in documents: # Identify training modules from content module_name = extract_module_name(doc.filename) # Create learning-optimized chunks chunks = doc.get_text_chunks( target_size=800, # Good for learning context tolerance=0.25 ) learning_chunks = [] for i, chunk in enumerate(chunks): learning_chunks.append({ 'content': chunk, 'module': module_name, 'sequence': i, 'learning_type': identify_learning_type(chunk), 'difficulty': assess_difficulty(chunk), 'prerequisites': extract_prerequisites(chunk) }) training_modules[module_name] = learning_chunks return training_modules def identify_learning_type(text): """Identify type of learning content""" text_lower = text.lower() if any(term in text_lower for term in ['example', 'case study', 'scenario']): return 'example' elif any(term in text_lower for term in ['concept', 'theory', 'principle']): return 'concept' elif any(term in text_lower for term in ['practice', 'exercise', 'hands-on']): return 'practice' else: return 'information' ``` ## Vector Database Integration ### Pinecone Knowledge Base ```python theme={null} import pinecone from cerevox import AsyncLexa async def build_pinecone_knowledge_base(documents): """Build knowledge base in Pinecone vector database""" # Initialize Pinecone pinecone.init(api_key="your-pinecone-key", environment="your-env") index = pinecone.Index("knowledge-base") async with AsyncLexa(api_key="your-api-key") as client: docs = await client.parse(documents) # Create knowledge-optimized chunks chunks = docs.get_all_text_chunks( target_size=700, # Good for knowledge retrieval tolerance=0.15 ) # Prepare vectors for Pinecone vectors = [] for i, chunk in enumerate(chunks): # Generate embedding (use your preferred embedding model) embedding = generate_embedding(chunk) # Create rich metadata metadata = { 'content': chunk, 'source': docs[i // len(chunks) * len(docs)].filename, 'domain': classify_domain(chunk), 'chunk_index': i, 'content_type': identify_content_type(chunk) } vectors.append({ 'id': f'kb_chunk_{i}', 'values': embedding, 'metadata': metadata }) # Batch upsert to Pinecone index.upsert(vectors=vectors) return f"Knowledge base: {len(vectors)} chunks indexed in Pinecone" async def query_knowledge_base(query: str, top_k: int = 5): """Query the Pinecone knowledge base""" # Generate query embedding query_embedding = generate_embedding(query) # Search Pinecone index = pinecone.Index("knowledge-base") results = index.query( vector=query_embedding, top_k=top_k, include_metadata=True ) # Format results knowledge_results = [] for match in results['matches']: knowledge_results.append({ 'content': match['metadata']['content'], 'source': match['metadata']['source'], 'domain': match['metadata']['domain'], 'relevance_score': match['score'] }) return knowledge_results ``` ## Performance for Knowledge Management **25 seconds** for 1000+ page knowledge corpus **98.5%** context preservation in chunking **95%** relevant results in top 5 matches ## Best Practices for Knowledge Bases ### Optimal Chunking Strategy ```python theme={null} # Different strategies for different knowledge types knowledge_chunking_strategies = { 'technical_docs': { 'target_size': 800, # Preserve technical context 'tolerance': 0.1 # Less flexibility for precision }, 'policies': { 'target_size': 600, # Complete policy sections 'tolerance': 0.2 # Allow for natural breaks }, 'faqs': { 'target_size': 300, # One Q&A per chunk 'tolerance': 0.15 # Maintain Q&A integrity }, 'procedures': { 'target_size': 500, # Complete procedural steps 'tolerance': 0.1 # Maintain step sequences } } async def smart_knowledge_chunking(documents, doc_type='general'): """Apply optimal chunking strategy based on document type""" strategy = knowledge_chunking_strategies.get(doc_type, { 'target_size': 600, 'tolerance': 0.15 }) async with AsyncLexa(api_key="your-api-key") as client: docs = await client.parse(documents) return docs.get_all_text_chunks( target_size=strategy['target_size'], tolerance=strategy['tolerance'] ) ``` ## Next Steps Build your first knowledge base in minutes Learn RAG implementation patterns Optimize for production knowledge systems Explore advanced parsing methods # Legal Research Source: https://docs.cerevox.ai/usecases/legal Transform legal document analysis with AI-powered parsing and precise extraction # Legal Document Analysis with Lexa Revolutionize legal research and document review with Lexa's enterprise-grade document parsing. Extract precise information from contracts, case files, and legal documents with unmatched accuracy. ## Why Lexa for Legal Practice? Extract clauses, terms, and citations with 99.5% accuracy SOC 2 certified with attorney-client privilege protection Analyze complex contracts and legal agreements Process thousands of discovery documents efficiently ## Supported Legal Documents * **Contracts & Agreements** (MSAs, NDAs, employment agreements) * **Court Filings** (pleadings, motions, briefs) * **Discovery Documents** (depositions, exhibits, correspondence) * **Case Law** (judicial opinions, legal precedents) * **Regulatory Filings** (SEC forms, compliance documents) * **Legal Research** (law journals, legal treatises) * **Patent Applications** and intellectual property documents ## Quick Start: Contract Analysis Extract key terms and clauses from legal contracts: ```python Contract Analysis theme={null} from cerevox import Lexa # Initialize client client = Lexa(api_key="your-api-key") # Parse legal contract documents = client.parse("service_agreement.pdf") contract = documents[0] # Extract key contract terms liability_clauses = contract.search_content("liability|indemnif") termination_terms = contract.search_content("termination|expire") payment_terms = contract.search_content("payment|fee|invoice") print(f"Contract: {contract.filename}") print(f"Pages: {contract.total_pages}") print(f"Liability clauses: {len(liability_clauses)}") print(f"Payment terms: {len(payment_terms)}") ``` ```python Async Batch Processing theme={null} import asyncio from cerevox import AsyncLexa async def analyze_contract_portfolio(): async with AsyncLexa(api_key="your-api-key") as client: # Process multiple contracts contracts = [ "msa_template.pdf", "nda_standard.docx", "employment_agreement.pdf" ] documents = await client.parse(contracts) contract_analysis = [] for doc in documents: analysis = { 'filename': doc.filename, 'pages': doc.total_pages, 'key_terms': extract_legal_terms(doc), 'risk_clauses': doc.search_content("risk|liable|penalty"), 'obligations': doc.search_content("shall|must|required") } contract_analysis.append(analysis) return contract_analysis def extract_legal_terms(document): """Extract common legal terms""" terms = [ "confidential", "proprietary", "intellectual property", "termination", "breach", "indemnification", "liability" ] found_terms = {} for term in terms: matches = document.search_content(term) found_terms[term] = len(matches) return found_terms # Run analysis results = asyncio.run(analyze_contract_portfolio()) ``` ## Advanced Legal Analysis ### Contract Clause Extraction Build a comprehensive contract analysis system: ```python theme={null} from cerevox import AsyncLexa import re from typing import Dict, List class ContractAnalyzer: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) # Define clause patterns self.clause_patterns = { 'termination': [ r'terminat[e|ion].*?(?:\.|;|\n)', r'end.*?agreement.*?(?:\.|;|\n)' ], 'liability': [ r'liabilit.*?(?:\.|;|\n)', r'indemnif.*?(?:\.|;|\n)' ], 'confidentiality': [ r'confidential.*?(?:\.|;|\n)', r'proprietary.*?(?:\.|;|\n)' ] } async def analyze_contract(self, contract_path: str) -> Dict: """Comprehensive contract analysis""" async with self.client: documents = await self.client.parse(contract_path) contract = documents[0] analysis = { 'contract_info': { 'filename': contract.filename, 'pages': contract.total_pages }, 'extracted_clauses': {}, 'risk_assessment': self._assess_risks(contract) } # Extract clauses by type for clause_type, patterns in self.clause_patterns.items(): clauses = [] for pattern in patterns: matches = re.findall(pattern, contract.content, re.IGNORECASE) clauses.extend(matches) analysis['extracted_clauses'][clause_type] = { 'count': len(clauses), 'text': clauses[:3] # First 3 matches } return analysis def _assess_risks(self, contract) -> Dict: """Assess legal risks in contract""" high_risk_terms = [ "unlimited liability", "personal guarantee", "liquidated damages", "penalty" ] risks = {} for term in high_risk_terms: matches = contract.search_content(term) if matches: risks[term] = { 'count': len(matches), 'severity': 'high' } return risks # Usage analyzer = ContractAnalyzer("your-api-key") analysis = await analyzer.analyze_contract("service_agreement.pdf") ``` ### Legal Document Search & RAG Build a legal research system with semantic search: ```python theme={null} from cerevox import AsyncLexa from typing import List, Dict class LegalRAGSystem: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) self.legal_corpus = [] async def ingest_legal_documents(self, document_paths: List[str]): """Ingest legal documents for search""" async with self.client: documents = await self.client.parse(document_paths) # Create searchable chunks optimized for legal content for doc in documents: chunks = doc.get_text_chunks( target_size=800, # Good for legal context tolerance=0.2 ) for i, chunk in enumerate(chunks): self.legal_corpus.append({ 'id': f"{doc.filename}_{i}", 'content': chunk, 'document': doc.filename, 'document_type': self._classify_document(doc.filename) }) return f"Ingested {len(self.legal_corpus)} legal document chunks" def search_legal_precedents(self, query: str, limit: int = 5) -> List[Dict]: """Search for legal precedents and relevant cases""" results = [] query_terms = query.lower().split() for chunk in self.legal_corpus: content_lower = chunk['content'].lower() # Score based on term frequency score = sum(1 for term in query_terms if term in content_lower) if score > 0: results.append({ 'content': chunk['content'], 'document': chunk['document'], 'relevance_score': score, 'document_type': chunk['document_type'] }) # Sort by relevance and return top results results.sort(key=lambda x: x['relevance_score'], reverse=True) return results[:limit] def _classify_document(self, filename: str) -> str: """Classify legal document type""" filename_lower = filename.lower() if any(term in filename_lower for term in ['contract', 'agreement']): return 'contract' elif any(term in filename_lower for term in ['case', 'opinion']): return 'case_law' elif any(term in filename_lower for term in ['brief', 'motion']): return 'court_filing' else: return 'legal_document' # Usage legal_rag = LegalRAGSystem("your-api-key") await legal_rag.ingest_legal_documents(["contracts.pdf", "case_law.pdf"]) results = legal_rag.search_legal_precedents("intellectual property") ``` ## Real-World Legal Use Cases ### Due Diligence Automation ```python theme={null} async def automated_due_diligence(target_company_docs): """Automate due diligence document review""" async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(target_company_docs) due_diligence_report = {} for doc in documents: # Extract critical information due_diligence_report[doc.filename] = { 'contract_terms': doc.search_content("term|clause|provision"), 'liabilities': doc.search_content("liabilit|debt|obligation"), 'intellectual_property': doc.search_content("patent|trademark"), 'litigation_risks': doc.search_content("lawsuit|litigation"), 'compliance_issues': doc.search_content("regulation|compliance") } return due_diligence_report ``` ### Discovery Document Processing ```python theme={null} async def process_discovery_documents(document_list: List[str]): """Process large volumes of discovery documents""" async with AsyncLexa(api_key="your-api-key") as client: BATCH_SIZE = 10 all_results = {} for i in range(0, len(document_list), BATCH_SIZE): batch = document_list[i:i + BATCH_SIZE] documents = await client.parse(batch) for doc in documents: key_info = { 'pages': doc.total_pages, 'communications': doc.search_content("email|letter|memo"), 'privileged_content': doc.search_content("attorney|counsel"), 'key_people': extract_people_names(doc) } all_results[doc.filename] = key_info return all_results ``` ## Performance for Legal Documents **45 seconds** for 500+ page legal brief **99.5%** accuracy on contract clauses **1000+ documents/hour** batch processing ## Security & Compliance Attorney-client privilege and work product protection maintained throughout processing * **Privileged Content**: Automatic detection and protection * **Data Isolation**: Client data never shared or used for training * **Audit Trail**: Complete processing logs for compliance * **Access Controls**: Role-based access with detailed permissions ## Next Steps Start processing legal documents in minutes Explore legal-specific parsing methods Learn production patterns for legal workflows Understand our security measures # Market Research Source: https://docs.cerevox.ai/usecases/market Extract actionable insights from market reports, surveys, and competitive intelligence # Market Research with Lexa Transform market research documents into actionable business intelligence. Lexa's advanced parsing extracts trends, metrics, and insights from complex market reports with precision. ## Why Lexa for Market Research? Parse complex market data tables with 99.8% accuracy Extract key metrics and growth trends automatically Aggregate insights across multiple research sources Extract competitor data and market positioning ## Market Research Applications * **Industry Reports** (Gartner, McKinsey, Forrester research) * **Market Surveys** (consumer insights, market sizing) * **Competitive Analysis** (competitor reports, product comparisons) * **Financial Research** (analyst reports, market forecasts) * **Customer Research** (satisfaction surveys, feedback analysis) * **Trend Analysis** (technology trends, market dynamics) * **Regulatory Research** (compliance reports, policy analysis) ## Quick Start: Market Analysis Extract key insights from market research documents: ```python Market Report Analysis theme={null} from cerevox import Lexa import re from typing import Dict, List # Initialize client client = Lexa(api_key="your-api-key") def analyze_market_report(report_path: str): """Extract key market insights from research report""" documents = client.parse(report_path) report = documents[0] # Extract market data market_insights = { 'report_name': report.filename, 'total_pages': report.total_pages, 'market_size': extract_market_size(report.content), 'growth_rates': extract_growth_rates(report.content), 'key_players': extract_companies(report.content), 'market_trends': extract_trends(report.content), 'forecasts': extract_forecasts(report.content), 'data_tables': len(report.tables) } return market_insights def extract_market_size(content: str) -> List[str]: """Extract market size mentions""" patterns = [ r'market size.*?\$?([\d,\.]+)\s*(billion|million|trillion)', r'valued at.*?\$?([\d,\.]+)\s*(billion|million|trillion)', r'market worth.*?\$?([\d,\.]+)\s*(billion|million|trillion)' ] market_sizes = [] for pattern in patterns: matches = re.findall(pattern, content, re.IGNORECASE) market_sizes.extend([f"${match[0]} {match[1]}" for match in matches]) return list(set(market_sizes)) def extract_growth_rates(content: str) -> List[str]: """Extract growth rate mentions""" growth_pattern = r'(?:growth|CAGR|increase).*?(\d+\.?\d*)%' matches = re.findall(growth_pattern, content, re.IGNORECASE) return [f"{match}%" for match in matches] def extract_companies(content: str) -> List[str]: """Extract company mentions (simplified)""" # Common company patterns company_patterns = [ r'[A-Z][a-z]+ (?:Inc|Corp|LLC|Ltd|Co)\.?', r'[A-Z][A-Za-z]+ [A-Z][a-z]+(?:\s+[A-Z][a-z]+)?' ] companies = [] for pattern in company_patterns: matches = re.findall(pattern, content) companies.extend(matches) # Remove duplicates and common false positives companies = list(set(companies)) return [c for c in companies if len(c) > 3][:10] # Top 10 def extract_trends(content: str) -> List[str]: """Extract market trends""" trend_keywords = [ 'artificial intelligence', 'ai', 'machine learning', 'cloud computing', 'digital transformation', 'remote work', 'sustainability', 'automation' ] found_trends = [] content_lower = content.lower() for trend in trend_keywords: if trend in content_lower: found_trends.append(trend) return found_trends def extract_forecasts(content: str) -> List[str]: """Extract forecast information""" forecast_patterns = [ r'forecast.*?(\d{4})', r'projected.*?(\d{4})', r'expected.*?(\d{4})', r'by (\d{4})' ] forecasts = [] for pattern in forecast_patterns: matches = re.findall(pattern, content, re.IGNORECASE) forecasts.extend(matches) return list(set(forecasts)) # Usage insights = analyze_market_report("market_research_2024.pdf") print(f"Market sizes found: {insights['market_size']}") print(f"Growth rates: {insights['growth_rates']}") print(f"Key players: {insights['key_players'][:5]}") ``` ```python Async Competitive Analysis theme={null} import asyncio from cerevox import AsyncLexa from typing import Dict, List import pandas as pd class CompetitiveIntelligence: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) async def analyze_competitive_landscape(self, competitor_reports: List[str]) -> Dict: """Analyze competitive landscape from multiple reports""" async with self.client: documents = await self.client.parse(competitor_reports) competitive_analysis = { 'market_overview': {}, 'competitor_profiles': [], 'market_positioning': {}, 'key_insights': [] } for doc in documents: # Extract competitor information competitor_data = self._extract_competitor_data(doc) competitive_analysis['competitor_profiles'].append(competitor_data) # Extract market positioning positioning = self._extract_market_positioning(doc) competitive_analysis['market_positioning'][doc.filename] = positioning # Extract key insights insights = self._extract_key_insights(doc) competitive_analysis['key_insights'].extend(insights) # Generate market overview competitive_analysis['market_overview'] = self._generate_market_overview( competitive_analysis['competitor_profiles'] ) return competitive_analysis def _extract_competitor_data(self, document) -> Dict: """Extract competitor-specific data""" content = document.content.lower() return { 'source': document.filename, 'pages': document.total_pages, 'revenue_mentions': len(document.search_content("revenue|sales")), 'product_mentions': len(document.search_content("product|service")), 'strategy_mentions': len(document.search_content("strategy|approach")), 'market_share': self._extract_market_share(document.content), 'strengths': self._extract_strengths_weaknesses(document.content, 'strength'), 'weaknesses': self._extract_strengths_weaknesses(document.content, 'weakness') } def _extract_market_positioning(self, document) -> Dict: """Extract market positioning information""" content = document.content positioning_keywords = { 'premium': ['premium', 'high-end', 'luxury', 'enterprise'], 'value': ['value', 'affordable', 'cost-effective', 'budget'], 'innovation': ['innovative', 'cutting-edge', 'advanced', 'next-generation'], 'established': ['established', 'mature', 'traditional', 'legacy'] } positioning_scores = {} content_lower = content.lower() for position, keywords in positioning_keywords.items(): score = sum(1 for keyword in keywords if keyword in content_lower) positioning_scores[position] = score return positioning_scores def _extract_key_insights(self, document) -> List[str]: """Extract key market insights""" # Look for insight indicators insight_chunks = document.search_content( "key insight|finding|conclusion|trend|opportunity|challenge" ) # Clean and filter insights cleaned_insights = [] for chunk in insight_chunks: if len(chunk) > 50 and len(chunk) < 500: # Reasonable insight length cleaned_insights.append(chunk.strip()) return cleaned_insights[:5] # Top 5 insights per document def _extract_market_share(self, content: str) -> List[str]: """Extract market share data""" import re market_share_patterns = [ r'market share.*?(\d+\.?\d*)%', r'(\d+\.?\d*)%.*?market share', r'holds.*?(\d+\.?\d*)%' ] shares = [] for pattern in market_share_patterns: matches = re.findall(pattern, content, re.IGNORECASE) shares.extend([f"{match}%" for match in matches]) return shares def _extract_strengths_weaknesses(self, content: str, type_: str) -> List[str]: """Extract strengths or weaknesses mentions""" if type_ == 'strength': keywords = ['strength', 'advantage', 'strong', 'leading', 'dominant'] else: keywords = ['weakness', 'challenge', 'limitation', 'struggle', 'weak'] mentions = [] content_lower = content.lower() for keyword in keywords: if keyword in content_lower: # Extract surrounding context import re pattern = f'.{{0,50}}{keyword}.{{0,50}}' matches = re.findall(pattern, content_lower) mentions.extend(matches) return mentions[:3] # Top 3 mentions def _generate_market_overview(self, competitor_profiles: List[Dict]) -> Dict: """Generate overall market overview""" total_documents = len(competitor_profiles) avg_revenue_mentions = sum( profile['revenue_mentions'] for profile in competitor_profiles ) / total_documents if total_documents > 0 else 0 avg_strategy_mentions = sum( profile['strategy_mentions'] for profile in competitor_profiles ) / total_documents if total_documents > 0 else 0 return { 'total_competitors_analyzed': total_documents, 'avg_revenue_mentions_per_report': avg_revenue_mentions, 'avg_strategy_mentions_per_report': avg_strategy_mentions, 'most_documented_competitor': max( competitor_profiles, key=lambda x: x['pages'] )['source'] if competitor_profiles else None } # Usage competitive_intel = CompetitiveIntelligence("your-api-key") competitor_reports = [ "competitor_a_analysis.pdf", "competitor_b_report.pdf", "market_leader_study.pdf" ] analysis = await competitive_intel.analyze_competitive_landscape(competitor_reports) print(f"Market Overview: {analysis['market_overview']}") print(f"Competitors Analyzed: {len(analysis['competitor_profiles'])}") print(f"Key Insights Found: {len(analysis['key_insights'])}") ``` ## Advanced Market Research Analytics ### Market Trend Analysis Track and analyze market trends across multiple reports: ```python theme={null} from cerevox import AsyncLexa import pandas as pd from collections import defaultdict from datetime import datetime import re class MarketTrendAnalyzer: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) self.trend_database = defaultdict(list) async def analyze_market_trends(self, research_documents: List[str], trend_categories: List[str] = None) -> Dict: """Analyze market trends across multiple research documents""" if trend_categories is None: trend_categories = [ 'artificial intelligence', 'cloud computing', 'sustainability', 'remote work', 'digital transformation', 'automation', 'cybersecurity', 'e-commerce', 'mobile technology' ] async with self.client: documents = await self.client.parse(research_documents) trend_analysis = { 'trends_by_document': {}, 'trend_frequency': defaultdict(int), 'trend_growth_indicators': {}, 'emerging_trends': [], 'declining_trends': [] } for doc in documents: doc_trends = self._extract_document_trends(doc, trend_categories) trend_analysis['trends_by_document'][doc.filename] = doc_trends # Update frequency counts for trend, data in doc_trends.items(): trend_analysis['trend_frequency'][trend] += data['mentions'] # Analyze trend growth indicators for trend in trend_categories: growth_data = self._analyze_trend_growth(documents, trend) trend_analysis['trend_growth_indicators'][trend] = growth_data # Identify emerging and declining trends trend_analysis['emerging_trends'] = self._identify_emerging_trends( trend_analysis['trend_frequency'] ) return trend_analysis def _extract_document_trends(self, document, trend_categories: List[str]) -> Dict: """Extract trend mentions from a single document""" doc_trends = {} content_lower = document.content.lower() for trend in trend_categories: mentions = content_lower.count(trend.lower()) if mentions > 0: # Look for growth indicators growth_indicators = self._find_growth_indicators(document.content, trend) # Extract trend context trend_context = self._extract_trend_context(document.content, trend) doc_trends[trend] = { 'mentions': mentions, 'growth_indicators': growth_indicators, 'context': trend_context[:3], # Top 3 contexts 'sentiment': self._assess_trend_sentiment(trend_context) } return doc_trends def _find_growth_indicators(self, content: str, trend: str) -> List[str]: """Find growth indicators for a specific trend""" growth_patterns = [ rf'{re.escape(trend)}.*?growing.*?(\d+\.?\d*)%', rf'{re.escape(trend)}.*?increased.*?(\d+\.?\d*)%', rf'{re.escape(trend)}.*?growth.*?(\d+\.?\d*)%', rf'(\d+\.?\d*)%.*?growth.*?{re.escape(trend)}' ] indicators = [] for pattern in growth_patterns: matches = re.findall(pattern, content, re.IGNORECASE) indicators.extend([f"{match}%" for match in matches]) return list(set(indicators)) def _extract_trend_context(self, content: str, trend: str) -> List[str]: """Extract contextual information about a trend""" import re # Find sentences containing the trend sentences = content.split('.') trend_contexts = [] for sentence in sentences: if trend.lower() in sentence.lower(): # Clean and add context cleaned_sentence = sentence.strip() if len(cleaned_sentence) > 20: # Skip very short sentences trend_contexts.append(cleaned_sentence) return trend_contexts def _assess_trend_sentiment(self, contexts: List[str]) -> str: """Assess sentiment around a trend (simplified)""" positive_words = ['growth', 'increase', 'opportunity', 'potential', 'strong', 'rising'] negative_words = ['decline', 'decrease', 'challenge', 'weak', 'falling', 'struggle'] positive_score = 0 negative_score = 0 for context in contexts: context_lower = context.lower() positive_score += sum(1 for word in positive_words if word in context_lower) negative_score += sum(1 for word in negative_words if word in context_lower) if positive_score > negative_score: return 'positive' elif negative_score > positive_score: return 'negative' else: return 'neutral' def _analyze_trend_growth(self, documents, trend: str) -> Dict: """Analyze growth patterns for a specific trend""" growth_data = { 'total_mentions': 0, 'documents_mentioning': 0, 'growth_rates': [], 'forecast_years': [] } for doc in documents: content = doc.content.lower() if trend.lower() in content: growth_data['documents_mentioning'] += 1 growth_data['total_mentions'] += content.count(trend.lower()) # Extract growth rates and forecasts growth_indicators = self._find_growth_indicators(doc.content, trend) growth_data['growth_rates'].extend(growth_indicators) # Extract forecast years forecast_pattern = rf'{re.escape(trend)}.*?(\d{{4}})' forecast_matches = re.findall(forecast_pattern, doc.content, re.IGNORECASE) growth_data['forecast_years'].extend(forecast_matches) return growth_data def _identify_emerging_trends(self, trend_frequency: Dict) -> List[str]: """Identify emerging trends based on frequency""" # Simple heuristic: trends mentioned frequently but not overwhelmingly frequencies = list(trend_frequency.values()) if not frequencies: return [] avg_frequency = sum(frequencies) / len(frequencies) emerging = [] for trend, freq in trend_frequency.items(): if avg_frequency * 0.5 <= freq <= avg_frequency * 1.5: emerging.append(trend) return emerging # Usage trend_analyzer = MarketTrendAnalyzer("your-api-key") research_docs = [ "tech_trends_2024.pdf", "market_forecast_report.pdf", "industry_analysis_q3.pdf" ] trend_analysis = await trend_analyzer.analyze_market_trends(research_docs) print(f"Most mentioned trends: {dict(list(trend_analysis['trend_frequency'].items())[:5])}") print(f"Emerging trends: {trend_analysis['emerging_trends']}") ``` ### Survey Data Analysis Process market survey documents and extract insights: ```python theme={null} class SurveyAnalyzer: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) async def analyze_survey_data(self, survey_documents: List[str]) -> Dict: """Analyze market survey documents""" async with self.client: documents = await self.client.parse(survey_documents) survey_analysis = { 'survey_metadata': [], 'response_patterns': {}, 'demographic_insights': {}, 'satisfaction_metrics': {}, 'key_findings': [] } for doc in documents: # Extract survey metadata metadata = self._extract_survey_metadata(doc) survey_analysis['survey_metadata'].append(metadata) # Analyze response patterns patterns = self._analyze_response_patterns(doc) survey_analysis['response_patterns'][doc.filename] = patterns # Extract satisfaction metrics satisfaction = self._extract_satisfaction_metrics(doc) survey_analysis['satisfaction_metrics'][doc.filename] = satisfaction # Extract key findings findings = self._extract_survey_findings(doc) survey_analysis['key_findings'].extend(findings) return survey_analysis def _extract_survey_metadata(self, document) -> Dict: """Extract survey metadata and basic info""" content = document.content # Extract sample size sample_patterns = [ r'sample size.*?(\d+)', r'(\d+).*?respondents', r'n\s*=\s*(\d+)' ] sample_size = None for pattern in sample_patterns: match = re.search(pattern, content, re.IGNORECASE) if match: sample_size = int(match.group(1)) break # Extract survey period date_patterns = [ r'conducted.*?(\d{4})', r'survey period.*?(\d{1,2}/\d{4})', r'fieldwork.*?(\d{4})' ] survey_period = None for pattern in date_patterns: match = re.search(pattern, content, re.IGNORECASE) if match: survey_period = match.group(1) break return { 'document': document.filename, 'pages': document.total_pages, 'sample_size': sample_size, 'survey_period': survey_period, 'tables_count': len(document.tables) } def _analyze_response_patterns(self, document) -> Dict: """Analyze survey response patterns""" content = document.content # Extract percentage responses percentage_pattern = r'(\d+\.?\d*)%' percentages = re.findall(percentage_pattern, content) percentages = [float(p) for p in percentages if float(p) <= 100] # Extract rating scales rating_patterns = [ r'(\d+)/10', r'(\d+) out of 10', r'rated (\d+)' ] ratings = [] for pattern in rating_patterns: matches = re.findall(pattern, content, re.IGNORECASE) ratings.extend([int(m) for m in matches]) return { 'percentage_responses': { 'count': len(percentages), 'average': sum(percentages) / len(percentages) if percentages else 0, 'distribution': self._categorize_percentages(percentages) }, 'rating_responses': { 'count': len(ratings), 'average': sum(ratings) / len(ratings) if ratings else 0, 'high_ratings': sum(1 for r in ratings if r >= 7) } } def _categorize_percentages(self, percentages: List[float]) -> Dict: """Categorize percentage responses""" categories = { 'high (70-100%)': sum(1 for p in percentages if p >= 70), 'medium (30-69%)': sum(1 for p in percentages if 30 <= p < 70), 'low (0-29%)': sum(1 for p in percentages if p < 30) } return categories def _extract_satisfaction_metrics(self, document) -> Dict: """Extract customer satisfaction metrics""" content = document.content.lower() satisfaction_keywords = { 'very_satisfied': ['very satisfied', 'extremely satisfied', 'highly satisfied'], 'satisfied': ['satisfied', 'pleased', 'happy'], 'neutral': ['neutral', 'neither', 'average'], 'dissatisfied': ['dissatisfied', 'unhappy', 'disappointed'], 'very_dissatisfied': ['very dissatisfied', 'extremely dissatisfied'] } satisfaction_scores = {} for category, keywords in satisfaction_keywords.items(): score = sum(content.count(keyword) for keyword in keywords) satisfaction_scores[category] = score return satisfaction_scores def _extract_survey_findings(self, document) -> List[str]: """Extract key survey findings""" finding_indicators = [ 'key finding', 'main finding', 'important finding', 'conclusion', 'result shows', 'data reveals' ] findings = [] for indicator in finding_indicators: matches = document.search_content(indicator) findings.extend(matches) return findings[:5] # Top 5 findings per document # Usage survey_analyzer = SurveyAnalyzer("your-api-key") survey_docs = [ "customer_satisfaction_2024.pdf", "market_research_survey.pdf", "brand_perception_study.pdf" ] survey_results = await survey_analyzer.analyze_survey_data(survey_docs) print(f"Surveys analyzed: {len(survey_results['survey_metadata'])}") print(f"Key findings: {len(survey_results['key_findings'])}") ``` ## Real-World Market Research Use Cases ### Industry Report Analysis ```python theme={null} async def analyze_industry_reports(report_paths: List[str]): """Comprehensive industry report analysis""" async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(report_paths) industry_analysis = { 'market_size_trends': {}, 'competitive_landscape': {}, 'technology_trends': {}, 'regulatory_changes': {}, 'investment_patterns': {} } for doc in documents: # Extract market sizing information market_data = extract_market_sizing(doc) industry_analysis['market_size_trends'][doc.filename] = market_data # Analyze competitive mentions competitive_data = analyze_competitive_mentions(doc) industry_analysis['competitive_landscape'][doc.filename] = competitive_data # Extract technology trends tech_trends = extract_technology_trends(doc) industry_analysis['technology_trends'][doc.filename] = tech_trends return industry_analysis def extract_market_sizing(document): """Extract market sizing data from document""" content = document.content # Market size patterns size_patterns = [ r'market.*?worth.*?\$?([\d,\.]+)\s*(billion|million|trillion)', r'industry.*?valued.*?\$?([\d,\.]+)\s*(billion|million|trillion)', r'revenue.*?\$?([\d,\.]+)\s*(billion|million|trillion)' ] market_sizes = [] for pattern in size_patterns: matches = re.findall(pattern, content, re.IGNORECASE) for match in matches: market_sizes.append({ 'value': match[0], 'unit': match[1], 'full_value': f"${match[0]} {match[1]}" }) return { 'market_sizes': market_sizes, 'count': len(market_sizes), 'tables_with_data': len(document.tables) } ``` ### Competitive Intelligence Dashboard ```python theme={null} class CompetitiveDashboard: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) self.competitor_data = {} async def build_competitive_dashboard(self, competitor_docs: Dict[str, List[str]]): """Build comprehensive competitive intelligence dashboard""" dashboard_data = { 'competitors': {}, 'market_positioning': {}, 'performance_metrics': {}, 'strategic_initiatives': {} } async with self.client: for competitor, doc_paths in competitor_docs.items(): documents = await self.client.parse(doc_paths) competitor_profile = { 'documents_analyzed': len(documents), 'total_pages': sum(doc.total_pages for doc in documents), 'revenue_data': self._extract_financial_data(documents), 'product_portfolio': self._extract_product_data(documents), 'market_position': self._assess_market_position(documents), 'recent_initiatives': self._extract_strategic_initiatives(documents) } dashboard_data['competitors'][competitor] = competitor_profile return dashboard_data def _extract_financial_data(self, documents): """Extract financial performance data""" financial_data = { 'revenue_mentions': 0, 'growth_rates': [], 'profit_margins': [], 'market_share': [] } for doc in documents: content = doc.content # Count revenue mentions financial_data['revenue_mentions'] += len(doc.search_content("revenue|sales")) # Extract growth rates growth_pattern = r'growth.*?(\d+\.?\d*)%' growth_matches = re.findall(growth_pattern, content, re.IGNORECASE) financial_data['growth_rates'].extend(growth_matches) # Extract market share share_pattern = r'market share.*?(\d+\.?\d*)%' share_matches = re.findall(share_pattern, content, re.IGNORECASE) financial_data['market_share'].extend(share_matches) return financial_data # Usage dashboard = CompetitiveDashboard("your-api-key") competitor_documents = { 'Company_A': ['company_a_annual_report.pdf', 'company_a_strategy.pdf'], 'Company_B': ['company_b_earnings.pdf', 'company_b_presentation.pdf'], 'Company_C': ['company_c_research.pdf', 'company_c_analysis.pdf'] } competitive_dashboard = await dashboard.build_competitive_dashboard(competitor_documents) ``` ## Performance for Market Research **15 seconds** for 200+ page market research report **95%** accuracy in trend and metric extraction **100+ reports/hour** for competitive intelligence ## Export and Visualization ### Export to Business Intelligence Tools ```python theme={null} import pandas as pd from cerevox import AsyncLexa async def export_for_bi_tools(research_documents: List[str], export_format: str = 'excel'): """Export market research data for BI tools""" async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(research_documents) # Structure data for BI consumption market_data = [] for doc in documents: # Extract structured insights insights = { 'document_name': doc.filename, 'analysis_date': datetime.now().strftime('%Y-%m-%d'), 'pages': doc.total_pages, 'data_tables': len(doc.tables), 'market_mentions': len(doc.search_content("market")), 'growth_mentions': len(doc.search_content("growth")), 'competitor_mentions': len(doc.search_content("competitor|competition")), 'trend_mentions': len(doc.search_content("trend|trending")) } market_data.append(insights) # Create DataFrame df = pd.DataFrame(market_data) # Export in requested format if export_format == 'excel': df.to_excel('market_research_analysis.xlsx', index=False) elif export_format == 'csv': df.to_csv('market_research_analysis.csv', index=False) elif export_format == 'json': df.to_json('market_research_analysis.json', orient='records') return df # Usage df = await export_for_bi_tools([ "q3_market_report.pdf", "competitor_analysis.pdf", "industry_trends.pdf" ], export_format='excel') print(f"Exported {len(df)} reports to Excel for BI analysis") ``` ## Next Steps Start analyzing market research in minutes Build market intelligence search systems Optimize for large-scale research analysis Explore advanced market data extraction # AI Model Training & Data Prep Source: https://docs.cerevox.ai/usecases/model Transform unstructured documents into ML-ready datasets with precision # AI Model Training & Data Preparation with Lexa Transform unstructured documents into high-quality training data for machine learning models. Lexa's precision parsing creates clean, structured datasets from complex documents. ## Why Lexa for AI/ML Workflows? Extract structured data with 99%+ accuracy for training Normalize data across multiple document formats Process thousands of documents for large datasets Extract features ready for ML pipelines ## ML Data Preparation Use Cases * **Text Classification Models** (document categorization, sentiment analysis) * **Named Entity Recognition** (extract entities from parsed content) * **Question-Answering Systems** (create Q\&A datasets from documents) * **Summarization Models** (extract summaries and key points) * **Information Extraction** (structured data from unstructured text) * **OCR Post-Processing** (clean and structure OCR output) * **Document Similarity** (create embeddings from parsed content) ## Quick Start: ML Dataset Creation Transform documents into ML-ready datasets: ```python Text Classification Dataset theme={null} from cerevox import Lexa import pandas as pd from sklearn.model_selection import train_test_split # Initialize client client = Lexa(api_key="your-api-key") # Process labeled documents for classification def create_classification_dataset(document_categories): """Create text classification dataset""" dataset = [] for category, doc_paths in document_categories.items(): documents = client.parse(doc_paths) for doc in documents: # Extract clean text chunks chunks = doc.get_text_chunks(target_size=512) for chunk in chunks: # Clean and prepare text cleaned_text = clean_text_for_ml(chunk) dataset.append({ 'text': cleaned_text, 'label': category, 'source': doc.filename, 'chunk_length': len(cleaned_text), 'word_count': len(cleaned_text.split()) }) # Convert to DataFrame df = pd.DataFrame(dataset) # Split into train/val/test train_df, temp_df = train_test_split(df, test_size=0.3, stratify=df['label']) val_df, test_df = train_test_split(temp_df, test_size=0.5, stratify=temp_df['label']) return { 'train': train_df, 'validation': val_df, 'test': test_df, 'stats': df.groupby('label').size().to_dict() } def clean_text_for_ml(text): """Clean text for ML training""" import re # Remove extra whitespace text = re.sub(r'\s+', ' ', text) # Remove special characters but keep punctuation text = re.sub(r'[^\w\s\.,!?;:-]', '', text) # Strip and return return text.strip() # Usage doc_categories = { 'financial': ['financial_report1.pdf', 'financial_report2.pdf'], 'legal': ['contract1.pdf', 'legal_doc1.pdf'], 'technical': ['manual1.pdf', 'spec1.pdf'] } dataset = create_classification_dataset(doc_categories) print(f"Training samples: {len(dataset['train'])}") print(f"Class distribution: {dataset['stats']}") ``` ```python Async ML Pipeline theme={null} import asyncio from cerevox import AsyncLexa import pandas as pd from typing import Dict, List class MLDataPreparationPipeline: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) async def prepare_training_data(self, data_sources: Dict[str, List[str]], task_type: str = 'classification'): """Prepare training data for different ML tasks""" async with self.client: all_data = [] for label, document_paths in data_sources.items(): print(f"Processing {label} documents...") documents = await self.client.parse(document_paths) # Task-specific data extraction if task_type == 'classification': data = self._extract_classification_features(documents, label) elif task_type == 'ner': data = self._extract_ner_features(documents, label) elif task_type == 'qa': data = self._extract_qa_pairs(documents, label) else: data = self._extract_general_features(documents, label) all_data.extend(data) return pd.DataFrame(all_data) def _extract_classification_features(self, documents, label): """Extract features for text classification""" features = [] for doc in documents: chunks = doc.get_text_chunks(target_size=400) for chunk in chunks: features.append({ 'text': chunk, 'label': label, 'source_filename': doc.filename, 'doc_pages': doc.total_pages, 'has_tables': len(doc.tables) > 0, 'text_length': len(chunk), 'sentence_count': len(chunk.split('.')), 'avg_word_length': self._avg_word_length(chunk) }) return features def _extract_ner_features(self, documents, category): """Extract features for Named Entity Recognition""" ner_data = [] for doc in documents: # Get smaller chunks for NER chunks = doc.get_text_chunks(target_size=200) for chunk in chunks: # Extract potential entities (simplified) entities = self._extract_entities(chunk, category) ner_data.append({ 'text': chunk, 'entities': entities, 'category': category, 'source': doc.filename }) return ner_data def _extract_qa_pairs(self, documents, domain): """Extract question-answer pairs from documents""" qa_pairs = [] for doc in documents: chunks = doc.get_text_chunks(target_size=800) for chunk in chunks: # Generate potential Q&A pairs questions = self._generate_questions(chunk) for question in questions: qa_pairs.append({ 'question': question, 'context': chunk, 'domain': domain, 'source': doc.filename, 'answer': self._extract_answer(question, chunk) }) return qa_pairs def _avg_word_length(self, text): """Calculate average word length""" words = text.split() if not words: return 0 return sum(len(word) for word in words) / len(words) def _extract_entities(self, text, category): """Extract named entities (simplified)""" import re entities = {} # Extract dates dates = re.findall(r'\d{1,2}/\d{1,2}/\d{4}|\d{4}-\d{2}-\d{2}', text) if dates: entities['dates'] = dates # Extract monetary amounts money = re.findall(r'\$[\d,]+\.?\d*', text) if money: entities['money'] = money # Extract percentages percentages = re.findall(r'\d+\.?\d*%', text) if percentages: entities['percentages'] = percentages return entities def _generate_questions(self, text): """Generate potential questions from text (simplified)""" questions = [] # Look for statement patterns that could be questions sentences = text.split('.') for sentence in sentences: sentence = sentence.strip() if len(sentence) > 20: # Skip very short sentences # Generate simple who/what/when questions if any(word in sentence.lower() for word in ['is', 'was', 'are', 'were']): questions.append(f"What {sentence.lower()}?") if any(word in sentence.lower() for word in ['company', 'organization']): questions.append(f"Who {sentence.lower()}?") return questions[:3] # Limit to 3 questions per chunk def _extract_answer(self, question, context): """Extract answer from context (simplified)""" # Very simplified answer extraction sentences = context.split('.') # Find sentence most likely to contain the answer question_words = set(question.lower().split()) best_sentence = "" best_overlap = 0 for sentence in sentences: sentence_words = set(sentence.lower().split()) overlap = len(question_words & sentence_words) if overlap > best_overlap: best_overlap = overlap best_sentence = sentence.strip() return best_sentence if best_sentence else context[:200] # Usage pipeline = MLDataPreparationPipeline("your-api-key") # Prepare classification data classification_sources = { 'positive_reviews': ['positive_reviews.pdf'], 'negative_reviews': ['negative_reviews.pdf'], 'neutral_reviews': ['neutral_reviews.pdf'] } df = await pipeline.prepare_training_data( classification_sources, task_type='classification' ) print(f"Dataset shape: {df.shape}") print(f"Label distribution:\n{df['label'].value_counts()}") ``` ## Advanced ML Data Workflows ### Feature Engineering Pipeline Extract ML-ready features from complex documents: ```python theme={null} from cerevox import AsyncLexa import pandas as pd import numpy as np from sklearn.feature_extraction.text import TfidfVectorizer from typing import List, Dict class DocumentFeatureExtractor: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) self.vectorizer = TfidfVectorizer(max_features=1000, stop_words='english') async def extract_comprehensive_features(self, documents: List[str]) -> pd.DataFrame: """Extract comprehensive features for ML models""" async with self.client: docs = await self.client.parse(documents) features = [] all_texts = [] for doc in docs: # Document-level features doc_features = { 'filename': doc.filename, 'total_pages': doc.total_pages, 'total_elements': doc.total_elements, 'table_count': len(doc.tables), 'image_count': len(doc.images) if hasattr(doc, 'images') else 0, 'file_size_kb': getattr(doc, 'file_size', 0) / 1024, } # Content-based features full_text = doc.content all_texts.append(full_text) # Text statistics doc_features.update({ 'char_count': len(full_text), 'word_count': len(full_text.split()), 'sentence_count': len(full_text.split('.')), 'paragraph_count': len(full_text.split('\n\n')), 'avg_word_length': np.mean([len(word) for word in full_text.split()]), 'punctuation_ratio': sum(1 for c in full_text if c in '.,!?;:') / len(full_text), 'uppercase_ratio': sum(1 for c in full_text if c.isupper()) / len(full_text), 'digit_ratio': sum(1 for c in full_text if c.isdigit()) / len(full_text) }) # Complexity features doc_features.update({ 'lexical_diversity': self._calculate_lexical_diversity(full_text), 'readability_score': self._calculate_readability(full_text), 'formality_score': self._calculate_formality(full_text) }) # Domain-specific features doc_features.update(self._extract_domain_features(full_text)) features.append(doc_features) # Create DataFrame df = pd.DataFrame(features) # Add TF-IDF features tfidf_matrix = self.vectorizer.fit_transform(all_texts) tfidf_df = pd.DataFrame( tfidf_matrix.toarray(), columns=[f'tfidf_{i}' for i in range(tfidf_matrix.shape[1])] ) # Combine all features final_df = pd.concat([df, tfidf_df], axis=1) return final_df def _calculate_lexical_diversity(self, text: str) -> float: """Calculate lexical diversity (unique words / total words)""" words = text.lower().split() if not words: return 0.0 return len(set(words)) / len(words) def _calculate_readability(self, text: str) -> float: """Calculate simplified readability score""" sentences = text.split('.') words = text.split() if not sentences or not words: return 0.0 avg_sentence_length = len(words) / len(sentences) avg_word_length = np.mean([len(word) for word in words]) # Simplified readability formula return max(0, 100 - (avg_sentence_length * 0.5) - (avg_word_length * 2)) def _calculate_formality(self, text: str) -> float: """Calculate formality score based on linguistic features""" formal_indicators = [ 'therefore', 'furthermore', 'moreover', 'consequently', 'accordingly', 'nevertheless', 'notwithstanding' ] informal_indicators = [ 'gonna', 'wanna', 'kinda', 'sorta', 'yeah', 'okay' ] text_lower = text.lower() formal_count = sum(1 for indicator in formal_indicators if indicator in text_lower) informal_count = sum(1 for indicator in informal_indicators if indicator in text_lower) total_indicators = formal_count + informal_count if total_indicators == 0: return 0.5 # Neutral return formal_count / total_indicators def _extract_domain_features(self, text: str) -> Dict: """Extract domain-specific features""" text_lower = text.lower() return { 'financial_terms': sum(1 for term in ['revenue', 'profit', 'loss', 'investment', 'roi'] if term in text_lower), 'legal_terms': sum(1 for term in ['contract', 'agreement', 'clause', 'liability', 'party'] if term in text_lower), 'technical_terms': sum(1 for term in ['system', 'process', 'method', 'algorithm', 'data'] if term in text_lower), 'medical_terms': sum(1 for term in ['patient', 'treatment', 'diagnosis', 'medical', 'health'] if term in text_lower) } # Usage extractor = DocumentFeatureExtractor("your-api-key") # Extract features from document collection document_paths = [ "financial_report.pdf", "legal_contract.pdf", "technical_manual.pdf" ] feature_df = await extractor.extract_comprehensive_features(document_paths) print(f"Feature matrix shape: {feature_df.shape}") print(f"Available features: {list(feature_df.columns)}") # Use features for ML model training from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split # Assuming you have labels X = feature_df.drop(['filename'], axis=1).fillna(0) y = ['financial', 'legal', 'technical'] # Your labels X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) model = RandomForestClassifier() model.fit(X_train, y_train) ``` ### Training Data Quality Pipeline Ensure high-quality training data: ```python theme={null} class TrainingDataQualityPipeline: def __init__(self, api_key: str): self.client = AsyncLexa(api_key=api_key) async def validate_training_data(self, document_paths: List[str], labels: List[str] = None) -> Dict: """Validate and analyze training data quality""" async with self.client: documents = await self.client.parse(document_paths) quality_report = { 'total_documents': len(documents), 'parsing_success_rate': len(documents) / len(document_paths), 'content_analysis': {}, 'recommendations': [] } # Analyze content quality text_lengths = [] error_documents = [] for i, doc in enumerate(documents): if not doc.content or len(doc.content.strip()) < 50: error_documents.append({ 'filename': doc.filename, 'issue': 'insufficient_content', 'content_length': len(doc.content) if doc.content else 0 }) else: text_lengths.append(len(doc.content)) # Content statistics if text_lengths: quality_report['content_analysis'] = { 'avg_content_length': np.mean(text_lengths), 'min_content_length': min(text_lengths), 'max_content_length': max(text_lengths), 'std_content_length': np.std(text_lengths), 'documents_with_errors': len(error_documents) } # Generate recommendations if error_documents: quality_report['recommendations'].append({ 'type': 'content_quality', 'message': f'{len(error_documents)} documents have insufficient content', 'action': 'Review and exclude documents with less than 50 characters' }) if text_lengths: length_std = np.std(text_lengths) length_mean = np.mean(text_lengths) if length_std > length_mean * 0.5: quality_report['recommendations'].append({ 'type': 'length_variance', 'message': 'High variance in document lengths detected', 'action': 'Consider chunking strategy or separate models for different document types' }) # Label balance analysis (if labels provided) if labels: label_counts = pd.Series(labels).value_counts() min_class_size = label_counts.min() max_class_size = label_counts.max() if max_class_size > min_class_size * 3: quality_report['recommendations'].append({ 'type': 'class_imbalance', 'message': 'Significant class imbalance detected', 'action': 'Consider data augmentation or stratified sampling', 'label_distribution': label_counts.to_dict() }) return quality_report def suggest_chunking_strategy(self, documents, target_model: str = 'bert') -> Dict: """Suggest optimal chunking strategy for ML model""" # Model-specific recommendations chunking_recommendations = { 'bert': {'target_size': 512, 'overlap': 50, 'tolerance': 0.1}, 'roberta': {'target_size': 512, 'overlap': 50, 'tolerance': 0.1}, 'longformer': {'target_size': 2048, 'overlap': 100, 'tolerance': 0.15}, 'bigbird': {'target_size': 2048, 'overlap': 100, 'tolerance': 0.15}, 'gpt': {'target_size': 1024, 'overlap': 0, 'tolerance': 0.1} } base_config = chunking_recommendations.get(target_model.lower(), { 'target_size': 512, 'overlap': 50, 'tolerance': 0.1 }) # Analyze document characteristics doc_lengths = [len(doc.content) for doc in documents if doc.content] avg_length = np.mean(doc_lengths) # Adjust recommendations based on document characteristics if avg_length < 1000: # Short documents - use smaller chunks base_config['target_size'] = min(base_config['target_size'], 256) elif avg_length > 10000: # Long documents - consider larger chunks if model supports if target_model.lower() in ['longformer', 'bigbird']: base_config['target_size'] = 4096 return { 'recommended_config': base_config, 'document_stats': { 'avg_length': avg_length, 'total_documents': len(documents), 'min_length': min(doc_lengths) if doc_lengths else 0, 'max_length': max(doc_lengths) if doc_lengths else 0 }, 'rationale': f"Optimized for {target_model} with average document length {avg_length:.0f} characters" } # Usage quality_pipeline = TrainingDataQualityPipeline("your-api-key") # Validate training data document_paths = ["doc1.pdf", "doc2.pdf", "doc3.pdf"] labels = ["class_a", "class_b", "class_a"] quality_report = await quality_pipeline.validate_training_data(document_paths, labels) print("Quality Report:", quality_report) # Get chunking recommendations docs = await AsyncLexa(api_key="your-api-key").parse(document_paths) chunking_strategy = quality_pipeline.suggest_chunking_strategy(docs, target_model='bert') print("Chunking Strategy:", chunking_strategy) ``` ## Real-World ML Use Cases ### Document Classification Model Training ```python theme={null} async def train_document_classifier(training_docs: Dict[str, List[str]]): """Complete pipeline for training document classifier""" # 1. Data preparation pipeline = MLDataPreparationPipeline("your-api-key") df = await pipeline.prepare_training_data(training_docs, task_type='classification') # 2. Feature extraction extractor = DocumentFeatureExtractor("your-api-key") all_docs = [doc for docs in training_docs.values() for doc in docs] features_df = await extractor.extract_comprehensive_features(all_docs) # 3. Combine data and features combined_df = df.merge(features_df, left_on='source_filename', right_on='filename') # 4. Train model from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import classification_report feature_columns = [col for col in combined_df.columns if col not in ['text', 'label', 'source_filename', 'filename']] X = combined_df[feature_columns].fillna(0) y = combined_df['label'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y) model = RandomForestClassifier(n_estimators=100, random_state=42) model.fit(X_train, y_train) # 5. Evaluate y_pred = model.predict(X_test) report = classification_report(y_test, y_pred) return { 'model': model, 'test_accuracy': model.score(X_test, y_test), 'classification_report': report, 'feature_importance': dict(zip(feature_columns, model.feature_importances_)) } ``` ### Named Entity Recognition Dataset ```python theme={null} async def create_ner_dataset(document_paths: List[str], entity_types: List[str] = None): """Create NER training dataset from documents""" if entity_types is None: entity_types = ['PERSON', 'ORG', 'GPE', 'DATE', 'MONEY'] async with AsyncLexa(api_key="your-api-key") as client: documents = await client.parse(document_paths) ner_dataset = [] for doc in documents: # Get sentences for NER training sentences = doc.content.split('.') for sentence in sentences: sentence = sentence.strip() if len(sentence) < 10: # Skip very short sentences continue # Extract entities (simplified - in production use spaCy/NER model) entities = extract_entities_for_ner(sentence, entity_types) if entities: # Only include sentences with entities ner_dataset.append({ 'text': sentence, 'entities': entities, 'source': doc.filename, 'length': len(sentence) }) return ner_dataset def extract_entities_for_ner(text: str, entity_types: List[str]) -> List[Dict]: """Extract entities in NER format (simplified)""" import re entities = [] # Date patterns date_pattern = r'\b\d{1,2}/\d{1,2}/\d{4}\b|\b\d{4}-\d{2}-\d{2}\b' for match in re.finditer(date_pattern, text): entities.append({ 'start': match.start(), 'end': match.end(), 'label': 'DATE', 'text': match.group() }) # Money patterns money_pattern = r'\$[\d,]+\.?\d*' for match in re.finditer(money_pattern, text): entities.append({ 'start': match.start(), 'end': match.end(), 'label': 'MONEY', 'text': match.group() }) # Simple person names (Title + Name pattern) person_pattern = r'\b(?:Mr|Ms|Mrs|Dr)\.?\s+[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*' for match in re.finditer(person_pattern, text): entities.append({ 'start': match.start(), 'end': match.end(), 'label': 'PERSON', 'text': match.group() }) return entities ``` ## Performance for ML Workflows **500+ docs/hour** for training data preparation **50+ features** extracted per document automatically **99.5%** clean data rate for model training ## Integration with ML Frameworks ### Hugging Face Integration ```python theme={null} from transformers import AutoTokenizer, AutoModelForSequenceClassification from cerevox import AsyncLexa async def prepare_for_huggingface(documents, model_name="bert-base-uncased"): """Prepare Lexa output for Hugging Face models""" tokenizer = AutoTokenizer.from_pretrained(model_name) async with AsyncLexa(api_key="your-api-key") as client: docs = await client.parse(documents) # Get chunks sized for the model max_length = tokenizer.model_max_length - 2 # Account for special tokens chunks = docs.get_all_text_chunks( target_size=max_length * 4, # Approximate character to token ratio tolerance=0.1 ) # Tokenize and prepare tokenized_data = [] for chunk in chunks: tokens = tokenizer( chunk, truncation=True, padding='max_length', max_length=max_length, return_tensors='pt' ) tokenized_data.append({ 'input_ids': tokens['input_ids'], 'attention_mask': tokens['attention_mask'], 'text': chunk }) return tokenized_data ``` ## Next Steps Start preparing ML datasets in minutes Learn embedding and RAG workflows Optimize for production ML pipelines Scale your ML data preparation # Authentication Source: https://docs.cerevox.ai/welcome/authentication All authentication methods for Cerevox Lexa API with copy-paste examples **Get your API key** from [cerevox.ai/lexa](https://cerevox.ai/lexa) - it's free to start. ## All Authentication Methods **Best for:** Production applications and general use ```bash theme={null} # Set your API key export CEREVOX_API_KEY="your-api-key-here" ``` ```python theme={null} from cerevox import Lexa # Client automatically reads from CEREVOX_API_KEY client = Lexa() # Test it works documents = client.parse(b"Test authentication") print("✅ Authenticated successfully!") ``` Most secure method - keeps keys out of your code **Best for:** Quick testing and prototyping ```python theme={null} from cerevox import Lexa # Pass API key directly client = Lexa(api_key="your-api-key-here") # For async operations from cerevox import AsyncLexa async def main(): async with AsyncLexa(api_key="your-api-key-here") as client: documents = await client.parse(["document.pdf"]) ``` Don't hardcode keys in production - use environment variables instead **Best for:** Local development ```bash .env theme={null} # Create .env file in your project root CEREVOX_API_KEY=your-api-key-here ``` ```python theme={null} import os from dotenv import load_dotenv from cerevox import Lexa # Load .env file load_dotenv() # Client reads from environment client = Lexa() ``` ```bash theme={null} # Install python-dotenv first pip install python-dotenv ``` **Best for:** Complex applications ```json config.json theme={null} { "cerevox": { "api_key": "your-api-key-here", "timeout": 120, "max_retries": 3 } } ``` ```python theme={null} import json from cerevox import Lexa with open("config.json") as f: config = json.load(f) client = Lexa( api_key=config["cerevox"]["api_key"], timeout=config["cerevox"]["timeout"], max_retries=config["cerevox"]["max_retries"] ) ``` ## Test Your Authentication Copy and run this to verify your setup works: ```python Quick Test theme={null} from cerevox import Lexa, LexaError def test_auth(): try: client = Lexa() # Test with sample content documents = client.parse(b"Authentication test content") if documents: print("✅ Authentication successful!") print(f"📄 Parsed: {documents[0].content}") return True except LexaError as e: print(f"❌ API Error: {e.message}") if "authentication" in e.message.lower(): print("💡 Check your API key at cerevox.ai/lexa") return False except Exception as e: print(f"❌ Error: {e}") return False # Run the test test_auth() ``` ```python Environment Check theme={null} import os def check_setup(): # Check if API key exists api_key = os.getenv("CEREVOX_API_KEY") if not api_key: print("❌ CEREVOX_API_KEY not found") print("💡 Set it: export CEREVOX_API_KEY='your-key'") return False if len(api_key) < 10: print("❌ API key seems invalid (too short)") return False print(f"✅ API key found: {api_key[:8]}...") return True check_setup() ``` ```python Async Test theme={null} import asyncio from cerevox import AsyncLexa, LexaError async def test_async_auth(): try: async with AsyncLexa() as client: documents = await client.parse(b"Async test content") if documents: print("✅ Async authentication works!") return True except LexaError as e: print(f"❌ API Error: {e.message}") return False asyncio.run(test_async_auth()) ``` ## Direct API Usage Using the REST API directly? Include your API key in the Authorization header: ```bash cURL theme={null} curl -X POST "https://data.cerevox.ai/v0/parse" \ -H "Authorization: Bearer your-api-key-here" \ -H "Content-Type: application/json" \ -d '{ "files": ["https://example.com/document.pdf"] }' ``` ```python requests theme={null} import requests headers = { "Authorization": "Bearer your-api-key-here", "Content-Type": "application/json" } response = requests.post( "https://data.cerevox.ai/v0/parse", headers=headers, json={"files": ["document.pdf"]} ) print(response.json()) ``` ```javascript fetch theme={null} const response = await fetch('https://data.cerevox.ai/v0/parse', { method: 'POST', headers: { 'Authorization': 'Bearer your-api-key-here', 'Content-Type': 'application/json' }, body: JSON.stringify({ files: ['document.pdf'] }) }); const data = await response.json(); console.log(data); ``` ## Common Issues & Solutions **Error:** `Authentication failed` or `Invalid API key` **Quick fixes:** ```python theme={null} # Check for extra spaces/quotes import os api_key = os.getenv('CEREVOX_API_KEY', '').strip() print(f"Key length: {len(api_key)}") print(f"First 8 chars: {api_key[:8]}") # Get a fresh key from cerevox.ai/lexa if needed client = Lexa(api_key=api_key) ``` **Error:** `No API key provided` **Quick fixes:** ```bash theme={null} # Check if it's set echo $CEREVOX_API_KEY # Set it (Linux/Mac) export CEREVOX_API_KEY="your-key" # Set it (Windows) set CEREVOX_API_KEY=your-key # Verify in Python import os; print(os.getenv('CEREVOX_API_KEY')) ``` **Error:** `Connection timeout` or network errors **Quick fixes:** ```python theme={null} # Increase timeout client = Lexa(timeout=180.0) # Check proxy settings if needed import os os.environ['HTTPS_PROXY'] = 'http://your-proxy:port' # Test with smaller content first client.parse(b"small test") ``` **Error:** `Insufficient permissions` or `Access denied` **Quick fixes:** * Check your account status at [cerevox.ai/lexa](https://cerevox.ai/lexa) * Verify your plan includes the features you're using * Check if you've hit rate limits (wait a minute and retry) * Contact [support@cerevox.ai](mailto:support@cerevox.ai) if issues persist ## Security Best Practices Keep API keys out of your code with environment variables Generate new keys every 90 days for enhanced security Watch for unusual API usage patterns in your dashboard Use API keys with only the permissions you need ## Production Setup Examples ```dockerfile theme={null} FROM python:3.9-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . # API key provided at runtime, not in image CMD ["python", "app.py"] ``` ```bash theme={null} # Run with API key docker run -e CEREVOX_API_KEY="your-key" my-lexa-app ``` ```yaml theme={null} name: Test Lexa on: [push] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: '3.9' - run: pip install cerevox - name: Test Lexa API env: CEREVOX_API_KEY: ${{ secrets.CEREVOX_API_KEY }} run: python test_lexa.py ``` ```yaml theme={null} apiVersion: v1 kind: Secret metadata: name: cerevox-secret type: Opaque stringData: api-key: your-api-key-here --- apiVersion: apps/v1 kind: Deployment metadata: name: lexa-app spec: template: spec: containers: - name: app image: my-lexa-app env: - name: CEREVOX_API_KEY valueFrom: secretKeyRef: name: cerevox-secret key: api-key ``` *** **Ready to parse?** Head to the [quickstart guide](/welcome/quickstart) to make your first API call, or check out [code examples](/examples/basic-usage). # Overview Source: https://docs.cerevox.ai/welcome/overview

The Data Layer for AI Agents
Precision retrieval get 70% smaller context — only relevant chunks, zero noise

**80% COST REDUCTION** Process 10x more requests with intelligent retrieval **99.5% ACCURACY** Flagship model quality at mini model cost **10x MORE REQUESTS** Smaller context windows = more throughput ## The Platform **Cerevox provides three powerful APIs for building AI agent data infrastructure:** **AI-powered search & Q\&A** → Semantic search across documents → Q\&A with source citations → 70% smaller context windows **Extract structured data** → 12+ file formats → Vector DB ready chunks → Cloud integrations **Enterprise operations** → Authentication & tokens → Usage tracking → User management ## Why Choose Cerevox? * **Precision RAG** - Only retrieve relevant chunks, eliminate noise * **70% smaller context** windows mean massive cost reduction * **99.5% accuracy match** to flagship models at mini model cost * **Smart chunking** optimized for semantic search and embeddings * **10x faster** than traditional solutions * **Native async support** across all APIs (Hippo, Lexa, Account) * **Enterprise-grade reliability** with automatic retries and error handling * **Batch processing** for thousands of documents * **Vector database ready** - Works with Pinecone, Weaviate, Chroma, etc. * **7+ cloud storage** integrations (S3, SharePoint, Google Drive, Box) * **Framework agnostic** - Django, Flask, FastAPI, LangChain * **Production ready** with comprehensive error handling and monitoring ## Get Started in 60 Seconds ```bash Installation theme={null} pip install cerevox ``` ```python Hippo - AI Q&A System theme={null} from cerevox import Hippo # Initialize Hippo client hippo = Hippo(api_key="your-api-key") # Create a knowledge base folder folder = hippo.create_folder("Product Documentation") # Upload documents hippo.upload_file(folder.id, "user-guide.pdf") hippo.upload_file(folder.id, "api-docs.pdf") # Create a chat session chat = hippo.create_chat(folder.id, "Technical Support") # Ask questions and get answers with citations answer = hippo.submit_ask(chat.id, "How do I authenticate users?") print(f"Answer: {answer.response}") print(f"Sources: {answer.sources}") ``` ```python Lexa - Document Parsing theme={null} from cerevox import Lexa # Initialize Lexa client client = Lexa(api_key="your-api-key") # Parse documents into structured data documents = client.parse(["invoice.pdf", "report.docx"]) # Get vector DB optimized chunks chunks = documents.get_all_text_chunks(target_size=500) print(f"Ready for embedding: {len(chunks)} chunks") ``` **Requirements:** Python 3.9+ • [Get your API key](https://cerevox.ai/lexa) from Cerevox ## Real-World Use Cases Build intelligent Q\&A over documents with source citations and 80% cost savings Create searchable knowledge bases with semantic search and RAG retrieval Query 10-K filings, reports, and financial statements with natural language Search contracts and legal documents with precision retrieval ## Next Steps Build your first RAG Q\&A system in 5 minutes Complete guide to semantic search and Q\&A Extract structured data from documents End-to-end RAG workflow examples *** **Ready to build?** Try our [Demo](https://cerevox.ai/lexa) or join our [Discord community](https://discord.gg/cerevox) for support. # Quickstart Source: https://docs.cerevox.ai/welcome/quickstart # Build Your First RAG Q\&A System in 5 Minutes 🦛 This quickstart gets you from zero to asking questions over your documents with AI-powered answers and source citations. **Requirements:** Python 3.9+ • 5 minutes of your time ## Step 1: Installation (30 seconds) ```bash Terminal theme={null} pip install cerevox ``` ```bash Alternative Methods theme={null} # Using conda conda install -c conda-forge cerevox # Using poetry poetry add cerevox # Using pipenv pipenv install cerevox ``` ## Step 2: Get API Key (30 seconds) Visit [cerevox.ai](https://cerevox.ai) and sign up for your free API key Save your API key - you'll need it in the next step ## Step 3: Authentication Setup (30 seconds) ```bash theme={null} export CEREVOX_API_KEY="your-api-key-here" ``` ```python theme={null} from cerevox import Hippo hippo = Hippo(api_key="your-api-key-here") ``` ```bash .env theme={null} CEREVOX_API_KEY=your-api-key-here ``` ```python theme={null} import os from dotenv import load_dotenv from cerevox import Hippo load_dotenv() hippo = Hippo() # Automatically reads from environment ``` ## Step 4: Build RAG Q\&A System (3 minutes) Copy and run this code to create your first AI Q\&A system: ```python Complete RAG Workflow theme={null} from cerevox import Hippo # Initialize Hippo (uses CEREVOX_API_KEY from environment) hippo = Hippo() # 1. Create a folder for your knowledge base folder = hippo.create_folder( name="Product Documentation", description="User guides and API docs" ) print(f"✅ Created folder: {folder.name} (ID: {folder.id})") # 2. Upload documents to the folder print("📤 Uploading documents...") # Upload from local file file1 = hippo.upload_file( folder_id=folder.id, file_path="user-guide.pdf" ) # Upload from URL file2 = hippo.upload_file_from_url( folder_id=folder.id, file_url="https://example.com/api-docs.pdf", file_name="api-docs.pdf" ) print(f"✅ Uploaded {file1.name} and {file2.name}") # 3. Create a chat session chat = hippo.create_chat( folder_id=folder.id, chat_name="Technical Support Q&A" ) print(f"✅ Created chat session: {chat.name}") # 4. Ask questions and get AI-powered answers! questions = [ "How do I authenticate users?", "What are the API rate limits?", "How do I handle errors?" ] for question in questions: print(f"\n❓ Question: {question}") answer = hippo.submit_ask( chat_id=chat.id, question=question ) print(f"💡 Answer: {answer.response}") print(f"📚 Sources: {len(answer.sources)} citations") # Show source details for source in answer.sources[:2]: # First 2 sources print(f" → {source.file_name} (Page {source.page_number})") print("\n🎉 Your RAG Q&A system is working!") ``` ```python Async Version (Recommended) theme={null} import asyncio from cerevox import AsyncHippo async def main(): async with AsyncHippo() as hippo: # 1. Create folder folder = await hippo.create_folder("Product Docs") # 2. Upload multiple files concurrently files = await asyncio.gather( hippo.upload_file(folder.id, "guide.pdf"), hippo.upload_file(folder.id, "docs.pdf") ) print(f"✅ Uploaded {len(files)} documents") # 3. Create chat chat = await hippo.create_chat(folder.id, "Support Q&A") # 4. Ask questions answer = await hippo.submit_ask( chat.id, "How do I get started?" ) print(f"💡 {answer.response}") print(f"📚 {len(answer.sources)} sources") asyncio.run(main()) ``` ```python Simple Q&A Example theme={null} from cerevox import Hippo hippo = Hippo() # Quick setup: Create folder, upload, create chat folder = hippo.create_folder("My Docs") hippo.upload_file(folder.id, "document.pdf") chat = hippo.create_chat(folder.id, "Q&A") # Ask your first question answer = hippo.submit_ask( chat_id=chat.id, question="What is the main topic of this document?" ) print(f"Answer: {answer.response}") print(f"Confidence: {answer.confidence_score}") # That's it! 80% cost reduction, flagship accuracy ``` **You're Ready!** 🎉 You've built your first RAG Q\&A system with 80% cost savings! ## What Just Happened? Folders organize your documents into searchable collections. Each folder becomes an isolated knowledge base for your AI agent. Hippo automatically processes and indexes your documents for semantic search. Supports PDFs, DOCX, PPTX, and more. Chat sessions maintain conversation context. Each chat remembers previous questions for follow-up queries. `submit_ask()` retrieves relevant chunks (70% smaller context) and generates answers with source citations. **80% cost reduction** vs. full document retrieval! ## Understanding the Cost Savings **Traditional RAG** * Sends entire documents * Large context windows * High token costs * Slower responses **Hippo RAG** * Only relevant chunks (70% smaller) * Precision retrieval * 80% cost reduction * 99.5% accuracy match ## Next Steps Complete guide to RAG capabilities Organize documents effectively Optimize answer quality Real-world implementation patterns Need to extract structured data from documents before using Hippo? Use **Lexa** for document parsing. ```python theme={null} from cerevox import Lexa client = Lexa() # Parse documents into structured data documents = client.parse(["contract.pdf"]) # Get vector DB optimized chunks chunks = documents.get_all_text_chunks(target_size=500) print(f"Ready for embedding: {len(chunks)} chunks") ``` Learn about document parsing capabilities ```python theme={null} from cerevox import Account account = Account() # Get account info info = account.get_account_info() print(f"Plan: {info.plan}") # Check usage usage = account.get_usage() print(f"API calls: {usage.total_requests}") ``` Manage authentication and usage ## Common Operations ```python theme={null} # Get all folders folders = hippo.get_folders() for folder in folders: print(f"{folder.name}: {folder.file_count} files") ``` ```python theme={null} # Get files in a folder files = hippo.get_files(folder_id=folder.id) for file in files: print(f"{file.name} - {file.status}") ``` ```python theme={null} # Get all chats for a folder chats = hippo.get_chats(folder_id=folder.id) for chat in chats: print(f"{chat.name}: {chat.message_count} messages") ``` ```python theme={null} # Get all Q&A history for a chat asks = hippo.get_asks(chat_id=chat.id) for ask in asks: print(f"Q: {ask.question}") print(f"A: {ask.response}\n") ``` ```python theme={null} # Delete a chat hippo.delete_chat(chat_id=chat.id) # Delete a file hippo.delete_file(file_id=file.id) # Delete a folder (and all its contents) hippo.delete_folder(folder_id=folder.id) ``` ## Having Issues? **Error:** `Authentication failed` or `Invalid API key` **Quick fixes:** * Double-check your API key from [cerevox.ai](https://cerevox.ai) * Verify `CEREVOX_API_KEY` environment variable is set correctly * Remove any extra spaces or quotes around the key * Try passing the key directly: `Hippo(api_key="your-key")` **Error:** File upload fails or times out **Quick fixes:** * Check file exists: `os.path.exists("your-file.pdf")` * Verify file permissions (must be readable) * For large files, increase timeout: `hippo.upload_file(folder_id, file, timeout=300)` * Supported formats: PDF, DOCX, PPTX, XLSX, TXT, HTML, CSV **Question:** How long does document processing take? **Answer:** * Small files (\< 10 pages): 10-30 seconds * Medium files (10-100 pages): 30-120 seconds * Large files (> 100 pages): 2-5 minutes * Use async API for better performance with multiple files * Files are queued and processed automatically **Question:** How can I improve answer quality? **Tips:** * Upload relevant documents only * Use descriptive folder and chat names * Ask specific, clear questions * Review source citations to verify answers * See [Q\&A Best Practices](/hippo/questions) for more tips *** **Ready to scale?** Check out our [RAG optimization guide](/guides/rag-optimization) or join the [Discord community](https://discord.gg/cerevox) for help.