All lessons T
Data Transformation beginner 8 min

Clean a List of IDs Before a Bulk Import

Sort and deduplicate identifiers before a bulk import creates duplicate work.

Scenario

A teammate assembled account IDs from three dashboards into one import box. The list has duplicates, a stray blank line, and no ordering — and the bulk import endpoint either rejects duplicate keys or, worse, processes them twice. Deduplicating by eye across a few dozen lines misses pairs that sit far apart; sort first and the duplicates become adjacent and obvious.

Broken input

acct_102
acct_099
acct_102

acct_120
acct_099

Goal

Use Line Sort / Unique / Deduplicate to produce one clean, sorted list of unique IDs that the import can consume without double-processing anything.

Tool sequence

  1. Open Line Sort / Unique / Deduplicate.
  2. Paste all five lines including the blank one — the tool should remove it, and seeing it removed confirms the cleanup worked.
  3. Apply sort and deduplicate together; sorting first is what makes the duplicate acct_102 and acct_099 entries collapse reliably.
  4. Count the output: five input lines became three unique IDs, which matches the two visible duplicates plus one blank line removed.
  5. If the IDs need to become a JSON array for the request body, feed the clean list into Query String Parser / JSON Converter or wrap it in your editor.

Checkpoint output

acct_099
acct_102
acct_120

Three unique IDs, sorted, no blanks — each account gets imported exactly once.

Common mistakes

  • Deduplicating before trimming whitespace, so acct_102 and acct_102 (trailing space) survive as two "different" IDs.
  • Changing ID casing during cleanup when the import treats IDs as case-sensitive.
  • Sorting a list whose original order had meaning — import priority, for example — without checking first.
  • Cleaning the list in the import box itself instead of somewhere the before/after can be compared.

Related tools and lessons