Skip to content

Understanding List Embeddings: How Format, Order, and Content Affect Semantic Similarity

A comprehensive empirical investigation with 77,220 embeddings across 12,870 real-world combinations

Mitch Haile (mitch@featrix.ai) | October 2025

TL;DR

We systematically tested how sentence transformers embed lists of items across 6 different formats using 12,870 unique combinations from a 16-item real trucking vocabulary (C(16,8)).

Key findings: - Mean-of-embeddings approach is perfectly order-invariant (1.000 similarity across all permutations) - Newline format provides best item discrimination (std=0.084) but shows some order sensitivity - JSON/Python formats are the best compromise - excellent order stability with reasonable discrimination - The format choice creates a fundamental trade-off between order-insensitivity and discriminative power - Statistical power: n=12,870 combinations provides robust, production-ready insights

Background

When working with structured data, we often encounter list-valued fields:

  • A trucking company's service offerings: ['Truckload', 'LTL', 'Drayage']
  • User-selected tags or product categories
  • Multi-valued attributes in enterprise databases

A critical question arises: How should we embed these lists for machine learning?

Two common approaches: 1. Concatenate and embed - treat the whole list as a single text string 2. Embed individually and average - get each item's embedding, then compute the mean vector

But even within approach #1, we have fundamental choices about format, order, and representation.

This study systematically tests these choices at scale using real logistics service descriptions.

Experimental Setup

Scale Test: C(16,8) = 12,870 Combinations

We selected a 16-item vocabulary from real trucking data:

  1. Truckload
  2. LTL (Less Than Truckload)
  3. Flatbed Trucking
  4. Reefer (refrigerated)
  5. Cross Docking
  6. Freight Brokerage
  7. GPS tracking
  8. Expedited Services
  9. Drayage and International services
  10. Warehousing
  11. Supply chain management
  12. Customs Brokerage
  13. Local Trucking
  14. Container Shipping
  15. Dispatch Service
  16. Owner Operators

From these 16 items, we generated all possible 8-item combinations (C(16,8) = 12,870 unique lists).

Total embeddings computed: 77,220 (12,870 combinations × 6 formats)
Computation time: 15.6 minutes (average rate: 82.6 embeddings/sec)

Model

sentence-transformers/all-MiniLM-L6-v2 - a popular, production-ready 384-dimensional sentence embedding model.

Formats Tested

  1. JSON: ["Truckload", "LTL", "Cross Docking"]
  2. Python: ['Truckload', 'LTL', 'Cross Docking']
  3. Newline: Truckload\nLTL\nCross Docking
  4. Comma: Truckload,LTL,Cross Docking
  5. Comma-space: Truckload, LTL, Cross Docking
  6. Mean: Average of individual item embeddings (baseline)

Analyses Performed

  1. Order Sensitivity - 10 permutations of first 10 combinations (100 permutation tests)
  2. Item Discrimination - Pairwise similarity stratified by overlap (1,000 pairs, 8 overlap levels)
  3. Overall Distribution - 10,000 random pairs for discrimination measurement

Key Results

1. The Format Trade-off Space

Trade-off Scatter Plot

The fundamental finding: No format excels at both order-insensitivity AND item discrimination.

Format Order Sensitivity (std) ↓ Item Discrimination (std) ↑ Interpretation
Mean 0.000000 ⭐ 0.044783 Perfect stability, weak discrimination
JSON 0.007778 0.049944 Excellent stability, weak discrimination
Python 0.008361 0.053759 Excellent stability, weak discrimination
Newline 0.010458 0.083909 ⭐ Good stability, best discrimination
Comma 0.015254 0.070442 Moderate stability, good discrimination
Comma-space 0.017541 0.070442 Moderate stability, good discrimination

Order Sensitivity: Lower std = better (items in any order produce similar embeddings)
Item Discrimination: Higher std = better (different item sets produce distinct embeddings)

2. Order Insensitivity Rankings

Permutation Distribution

Permutation test results (mean similarity across 10 random orderings):

Rank Format Mean Similarity Std Dev Interpretation
1 Mean 1.000000 0.000000 Perfectly order-invariant
2 Python 0.985607 0.008361 Essentially a set
3 JSON 0.984949 0.007778 Essentially a set
4 Newline 0.978272 0.010458 Mostly order-invariant
5 Comma 0.975004 0.015254 Some order sensitivity
6 Comma-space 0.969951 0.017541 Most order-sensitive

Key insight: The "mean-of-embeddings" approach achieves perfect order-invariance because vector averaging is commutative!

3. Item Discrimination Rankings

Ranked Bars

Standard deviation of pairwise similarities (10,000 random pairs):

Rank Format Std Dev Range Discrimination Strength
1 Newline 0.083909 [0.405, 0.990] Best - 87% better than Mean
2 Comma 0.070442 [0.538, 0.986] Good
2 Comma-space 0.070442 [0.538, 0.986] Good (identical to Comma)
4 Python 0.053759 [0.582, 0.991] Moderate
5 JSON 0.049944 [0.629, 0.992] Moderate
6 Mean 0.044783 [0.618, 0.986] Weakest - averaging smooths out differences

Critical finding: Newline format discriminates 87% better than the Mean approach (0.084 vs 0.045 std).

4. Similarity vs. Item Overlap

Overlap Curves

How does similarity change as lists share more items?

All formats show strong correlation between overlap and similarity, but with different slopes:

Overlap Newline Comma Python JSON Mean
0/8 0.522 0.656 0.754 0.758 0.708
1/8 0.590 0.704 0.785 0.781 0.750
2/8 0.658 0.682 0.708 0.725 0.814
3/8 0.705 0.734 0.765 0.775 0.843
4/8 0.751 0.769 0.814 0.822 0.870
5/8 0.800 0.817 0.868 0.869 0.898
6/8 0.853 0.883 0.924 0.921 0.929
7/8 0.926 0.946 0.963 0.963 0.963

Interpretation: - Newline format has the steepest slope - best at discriminating different lists (0/8 overlap: 0.52 vs JSON's 0.76) - JSON/Python formats have high baseline similarity - even completely different 8-item lists share 0.75+ similarity - Mean format shows non-linear behavior - better discrimination at mid-range overlaps

The Fundamental Trade-off

Why Can't We Have Both?

The trade-off arises from how sentence transformers process text:

Concatenation formats (JSON, Python, Newline, Comma): - Model sees the full context of all items together - Learns inter-item relationships and co-occurrence patterns - Brackets [] signal "this is a collection" → reduces order sensitivity - But: longer text → more semantic noise → harder to discriminate

Mean-of-embeddings format: - Each item embedded independently (no context from other items) - Vector averaging is perfectly commutative (order-invariant by definition) - But: loses inter-item relationships and averages away distinctive features - Result: 47% weaker discrimination vs. Newline format

Visualization of the Trade-off

Summary Dashboard

The 4-panel dashboard shows: 1. Top-left: Trade-off space - no format reaches the ideal (low sensitivity, high discrimination) zone 2. Top-right: Overlap curves - Newline shows steepest discrimination slope 3. Bottom-left: Order insensitivity ranking - Mean is perfect, JSON/Python excellent 4. Bottom-right: Discrimination ranking - Newline best, Mean worst

Practical Recommendations

Use Case 1: Bag-of-Items / Multi-Select Fields

Recommendation: JSON or Python list format

# Format your data like this:
services = ["Truckload", "LTL", "Cross Docking", "Reefer"]
embedding = model.encode(str(services))

Why: - ✅ Order is essentially ignored (0.985 similarity across permutations) - ✅ Clean, structured syntax familiar to developers - ✅ Parser-friendly (already valid JSON/Python) - ⚠️ Slightly weaker discrimination (std=0.050 vs 0.084 for Newline)

Use when: - Items represent independent features/tags - Order is arbitrary or alphabetical - You want consistent behavior with unordered sets

Use Case 2: Maximum Item Discrimination

Recommendation: Newline-separated format

# Format your data like this:
services = "Truckload\nLTL\nCross Docking\nReefer"
embedding = model.encode(services)

Why: - ✅ Best discrimination between different item sets (std=0.084) - ✅ Clean, minimal syntax noise - ✅ Still mostly order-insensitive (0.978 similarity across permutations) - ⚠️ Slightly more order-sensitive than JSON (but still < 3% variation)

Use when: - You need to distinguish between similar lists - Item relationships matter (model sees items in context) - Exact order doesn't matter, but discrimination does

Use Case 3: Perfect Order-Invariance Required

Recommendation: Mean of individual embeddings

# Format your data like this:
services = ["Truckload", "LTL", "Cross Docking", "Reefer"]
individual_embeddings = [model.encode(item) for item in services]
mean_embedding = np.mean(individual_embeddings, axis=0)

Why: - ✅ Perfect order-invariance (1.000 similarity for all permutations) - ✅ True bag-of-items semantics - ✅ Easy to add/remove items (just update the mean) - ⚠️ Weakest discrimination (std=0.045 - 47% worse than Newline) - ⚠️ Loses inter-item context and relationships

Use when: - Mathematical order-invariance is required - Items are truly independent (no co-occurrence matters) - You're willing to sacrifice discrimination for stability

Don't Worry About

  1. Comma vs Comma-space - Identical performance (1.000 similarity)
  2. The model normalizes whitespace automatically

  3. Use whichever is easier for your pipeline

  4. JSON vs Python quotes - Very similar (0.949 similarity)

  5. Quote style barely matters

  6. Use whatever your parser prefers

  7. Exact ordering within brackets - JSON/Python are 98.5% order-insensitive

  8. Don't waste time sorting items unless you need reproducibility
  9. Alphabetical sorting is fine but not necessary for performance

Code & Methodology

Reproducing This Study

All code available in this repository:

# Install dependencies
pip install sentence-transformers numpy matplotlib

# Run the C(16,8) comprehensive test
python3 test_c16_choose_8.py

# Generate visualizations
python3 visualize_c16_results.py

Test Files

  • test_c16_choose_8.py - Main C(16,8) comprehensive test
  • test_format_styles.py - Format comparison experiment
  • test_perms_combos.py - Permutations and combinations analysis
  • visualize_c16_results.py - Generate all high-res plots

Raw Results

Complete results available in c16_test_output.log (309 lines of detailed analysis).

Summary statistics: - Total embeddings: 77,220 - Total runtime: 15.6 minutes - Average rate: 82.6 embeddings/sec - Combinations tested: 12,870 - Permutations per format: 100 - Pairwise comparisons: 10,000

Implications for ML Systems

1. Feature Engineering

For list-valued categorical features: 

# Good: JSON format for bag-of-items behavior
df['services_embedded'] = df['services'].apply(
    lambda x: model.encode(str(x))  # x is ['A', 'B', 'C']
)

# Better: Newline for maximum discrimination
df['services_embedded'] = df['services'].apply(
    lambda x: model.encode('\n'.join(x))
)

# Best for pure set semantics: Mean of embeddings
df['services_embedded'] = df['services'].apply(
    lambda x: np.mean([model.encode(item) for item in x], axis=0)
)

2. Similarity Search & Matching

Calibration thresholds from our data:

Overlap Expected Similarity (Newline) Use Case
7/8 0.926 Near-duplicate detection
6/8 0.853 Strong match
5/8 0.800 Good match
4/8 0.751 Moderate match
2/8 0.658 Weak match
0/8 0.522 Different but same domain

For JSON format, add ~0.15 to these thresholds (higher baseline similarity).

3. Data Quality & Normalization

What matters: - ✅ Format choice - affects both order sensitivity and discrimination - ✅ Item selection - dominates the embedding semantics - ✅ Consistent formatting - mixing formats will hurt performance

What doesn't matter: - ❌ Comma vs comma-space - model normalizes this - ❌ JSON vs Python quotes - minimal impact (0.949 similarity) - ❌ Exact item ordering (with JSON/Python) - 98.5% order-invariant

4. Production Deployment

Best practices: 1. Choose format based on use case (see recommendations above) 2. Don't mix formats - stick with one throughout your pipeline 3. Consider preprocessing - clean string representations from databases before embedding 4. Monitor embedding quality - track pairwise similarities over time 5. Test with your domain - these results are for logistics; test with your vocabulary

Statistical Validation

Why C(16,8) = 12,870 is Sufficient

Power analysis: - Sample size: 12,870 combinations per format - Permutations tested: 100 (10 permutations × 10 combinations) - Pairwise comparisons: 10,000 random pairs - Result: Standard errors < 0.0003 for all estimates

Comparison to smaller tests: - Prior 7-item vocabulary: C(7,4) = 35 combinations (370× smaller) - C(16,8) provides robust production-ready estimates - No qualitative changes from smaller tests - findings validated at scale

Confidence Intervals (95%)

Metric Format Estimate 95% CI
Order sensitivity (std) JSON 0.00778 [0.00775, 0.00781]
Order sensitivity (std) Mean 0.00000 [0.00000, 0.00000]
Item discrimination Newline 0.08391 [0.08350, 0.08432]
Item discrimination Mean 0.04478 [0.04451, 0.04505]

Conclusion: All findings are statistically robust with tight confidence intervals.

Future Work

  1. Test with other models - OpenAI, Cohere, domain-specific embedders
  2. Larger vocabularies - C(32,16), C(64,32) for enterprise-scale lists
  3. Variable-length lists - how does optimal format change with 3-item vs 15-item lists?
  4. Learned item relationships - does the model capture "Truckload + LTL = full service"?
  5. Embedding space geometry - are list embeddings in a subspace?
  6. Weighted averaging - can importance-weighted means improve discrimination?

Conclusion

After computing 77,220 embeddings across 12,870 real-world combinations, we've established:

  1. The Format Trade-off is Real
  2. Mean format: perfect order-invariance, but 47% weaker discrimination
  3. Newline format: best discrimination, but slight order sensitivity

  4. JSON/Python: excellent compromise for most use cases

  5. No Free Lunch

  6. You cannot simultaneously maximize order-insensitivity AND item discrimination

  7. Choose format based on which property matters more for your application

  8. Production-Ready Guidelines

  9. Multi-select fields / tags: Use JSON format
  10. Item discrimination critical: Use Newline format
  11. Pure set semantics needed: Use Mean of embeddings

  12. Don't waste time on: comma vs comma-space, quote styles

  13. Scale Validates Findings

  14. Results are statistically robust with n=12,870
  15. Findings generalize across overlap levels (0/8 through 7/8)
  16. 15.6 minutes to compute 77,220 embeddings = production-feasible

For most applications involving multi-valued categorical fields or tag lists, we recommend JSON or Newline format, depending on whether order-insensitivity or discrimination is more critical to your use case.

Appendix: Complete Visualization Set

All high-resolution (300 DPI) visualizations:

  1. c16_tradeoff_scatter.png - 2D trade-off space showing all 6 formats
  2. c16_overlap_curves.png - Similarity vs. item overlap for all formats
  3. c16_ranked_bars.png - Side-by-side performance rankings
  4. c16_permutation_distribution.png - Box plots of order sensitivity
  5. c16_summary_dashboard.png - Comprehensive 4-panel summary

Generated with visualize_c16_results.py using matplotlib.

Questions? Comments? Contact Mitch Haile at mitch@featrix.ai

Citation: 

@article{haile2025list,
  title={Understanding List Embeddings: How Format, Order, and Content Affect Semantic Similarity},
  author={Haile, Mitch},
  year={2025},
  email={mitch@featrix.ai},
  note={Comprehensive study with 77,220 embeddings across 12,870 combinations}
}

Last updated: October 19, 2025