Issue

When a Segment Engage audience is connected to a streaming destination like Amazon Kinesis Firehose, the exported audience files or streams appear to have duplicate members. This causes downstream duplication issues because each user appears in the destination more than once, sometimes on a daily basis, even when ID Sync is configured to send a single identifier.

Product

Segment 

Environment

Segment Console

Cause

There are two underlying technical causes for this apparent duplication in streaming destinations:

1. Incorrect Ingestion Mapping: Even when ID Sync successfully restricts the payload to a specific identifier (like customer_id), the destination receives a raw JSON document where the requested identifier is nested within the traits or properties object. Downstream ingestion scripts often default to extracting the root-level userId (which may be a constantly changing UUID or device ID) instead of the specific mapped trait, creating what looks like duplicate rows for the same user.
2. Audience Exit Events: Dynamic audiences evaluate users continuously. When a user qualifies for an audience, Segment sends an Identify or Track event with a boolean flag set to true. When the user no longer meets the criteria, Segment sends a follow-up event with the flag set to false. Append-only streaming logs record both of these events as new rows. If the downstream system does not check the true/false flag, it interprets the exit event as a duplicate entry.

Resolution

To resolve the duplication on the destination side, instruct the data team managing the ingestion pipeline to adjust their scripts with the following steps:

1. Navigate to the Segment Workspace and use the Source Debugger for the Engage space to inspect a sample raw JSON payload.
2. Identify the exact nested path of the desired identifier (for example, locate traits.customer_id instead of the root-level userId).
3. Update the downstream ingestion script (e.g., AWS ETL, Glue job, or Lambda function) to extract the identifier from that specific nested object path.
4. Update the downstream script's logic to check the boolean value of the audience key trait.
5. Configure the system to ignore or drop rows where the audience boolean flag is false to prevent processing audience exit events as duplicate entries.

Additional Information 

• By default, an Engage Audience generates and sends one event for every identifier attached to a user's profile. ID Sync is highly recommended to explicitly restrict which identifiers Segment sends downstream.
• Unlike API-based destinations that update user states in-place, streaming destinations (like Amazon Kinesis Firehose) are append-only. Every profile update or audience exit emitted by Segment will result in a new row being appended to the stream.