Use the Dead Letter Strategy
- Last UpdatedMar 27, 2025
- 2 minute read
To use the deadletter
strategy, you must:
- Add two properties to the
application.conf
file. - Create an index layer for archiving messages that encounter an error.
Dead Letter Properties in application.conf
The two properties you must add to the application.conf
file are as follows:
deadletter.hrn
deadletter.layer
Both properties go under error
settings. For more information, see Configure Your Application.
Dead Letter Index Layer
The deadletter
error handling strategy archives messages that encounter an error to an index layer. This special index layer must have the following settings:
Content Type must be application/x-avro-binary.
Content Encoding must be uncompressed.
There must be four indexing attributes with the following names and settings:
- A
timewindow
type attribute with the name**indexUploadTime**
. You can select the desiredduration
. This attribute stores the timestamp of the index upload, truncated by the duration value. - A
string
type attribute with the name**realm**
. This attribute stores the realm of the index layer where messages that were successfully processed are indexed. - A
string
type attribute with the name**catalogId**
. This attribute stores thecatalogId
of the index layer where messages that were successfully processed are indexed. - A
string
type attribute with name**layerId**
. This attribute stores thelayerId
of the index layer where messages that were successfully processed are indexed.
- A
You can select any TTL setting.
If the special index layer does not follow these settings, the pipeline will fail to initialize.
Custom Dead Letter Transformation
For deadletter
error strategy, you can provide a custom implementation of the method transformForDeadLetter(message)
in the UDF interface. This allows you to modify the contents of the message
field in the deadletter message schema, shown below, before it is archived to the deadletter index layer.
Example
@Override
public byte[] transformForDeadLetter(byte[] message) {
return (new String(message) + "this message has been modified").getBytes();
}
Note
transformForDeadLetter
is an optional API. If you don't provide any implementation, the original message will be archived in deadletter index layer.- If your implementation of
transformForDeadLetter
API causes an error, the original message will be archived in deadletter index layer.
Dead Letter Messages
For deadletter
error strategy, the Data Archiving Library aggregates and archives the messages that encounter an error scenario in a User-Defined Function's implementation.
Messages are archived with following metadata:
messageId
ingestionTime
processingTime
The Data Archiving Library archives the deadletter messages in the Avro format. The Avro Schema DeadLetterMessage
is:
You can use this Avro schema to reprocess or deserialize the deadletter messages stored in the deadletter index layer.