Introduction to the Quick Processor

The Quick Processor allows you to create filters and modify records using JavaScript. In this article, we show how to create a Quick Processor and how to access, filter, and modify records. A Quick Processor is always linked to a view. This means that a Quick Processor is saved when a view is saved.

Usage Scenarios

Transform records (Data Transformation)

Use the Quick Processor to modify existing attributes of your data objects or add new ones that can be used for filtering using the attribute filters or sorting.

Data Correction

In combination with the Multi-Select functionality, you can use the Quick Processor to correct data, select it via Multi-Select and send it to a stream again.
See the following video to learn how to correct records using the Quick Processor.

Calculation of results based on multiple data sets (e.g., moving average or grouping)

Use the state store of the Quick Processor to store calculation results across multiple executions (e.g., to calculate a moving average) or to group data sets.

Complex filter logic

With JavaScript you are capable of modeling even the most complicated business logic. In combination with the attribute filters, you can quickly create useful reports for data analytics or business users.

Hello World

This code randomly picks (filters) a record out of your stream and sets the record's value to "Hello World!".

1. if ( Math.random() < 0.5) // Filter a record randomly.
   
2. return false; // If false, the record is not displayed.
   
3. rec.value = "Hello World!"; // Sets a new value for each record
   
4. return rec; // Returns the new record with "Hello World!".

Transform data

Overwrite, delete or calculate new values for message value attributes and keys.

1. // Overwrite the record's key and add a new attribute
2. // to the record's value.
3. rec.value.myAttribute = "Hello World!";
4. rec.key = "My Key";
5. ‍
6. // Delete an attribute
7. delete rec.value.obsoleteAttr;
8. 
   
9. return rec;

Filter data

Create complex filters with full JavaScript.

1. // No filtering (all records are let through)
2. ‍return true;
3. ‍
4. // View only records that are from January (0)
5. return new Date(rec.value.deliveryTime).getMonth() != 0;

Store data (aggregates)

Calculate and temporarily store data to aggregate or group data.

1. // Increment stored value; Init with 1.
2. let count = (restore() == null) ? 1 : restore() + 1;
3. 
   
4. // Assign current count to an attribute
5. rec.value.Count = count;
6. ‍
7. // Store current count‍
8. store(count);
9. 
   
10. return rec;

Accessible attributes

You can access the attributes in code as follows:

1. rec.value       // record's decoded value
2. rec.key         // record's decoded key
3. rec.timestamp   // record's timestamp
4. rec.topic       // the topic
5. rec.partition   // record's partition / shard
6. rec.offset      // the offset (Apache Kafka only)
7. rec.headers     // record's headers (Apache Kafka only)
8. 
   
9. // If the value or key is a JSON object with a field "myAttr":
10. rec.value.myAttr
11. rec.key.myAttr

Tip: Quickly copy the path of an attribute to the clipboard

Writing out the path of an attribute can be annoying. Therefore, select a record in the record list and click on the corresponding attribute in the detail view. This opens the filter bubble. Click on the name of the attribute on the left. You now have the complete path in the clipboard.

State and stateful calculations

The Quick Processor is executed for each individual data record. To group records or perform calculations based on a previous data set or result, a state can be saved in the state store.

A state is saved with the store(any) command. The restore():any command can be used to load the state from the state memory into a variable, for example. To initialize a state memory (e.g. to set an initial value), the command hasStore():boolean can be used to query whether a state already exists.

Moving average example

This code gets the value of the record and adds it to the value of the previous record that was stored. It then divides the result by two to get the moving average and returns a JSON object with the moving average and the current temperature.

1. var fahrenheit = rec.value;
2. if(!hasStore())
3. store(fahrenheit); // Init state with current temperature
4. // Divide the previous average by 2;
5. store((restore() + fahrenheit) / 2);
6. // Setting the calculated and the original value as the new record's value
7. rec.value = { "Average":restore(), "Fahrenheit": fahrenheit };
8. // Returning the new record with the new value
9. return rec;

Tip: Store JSON Objects or arrays if you need to store multiple results (or records).

You can also store entire JSON objects or arrays instead of a single value. Please make sure to delete the objects as soon as they are no longer needed to avoid filling up the memory during a request.

Accessing header values

The following snippet finds the first header with the name "MyHeader" and converts it into a string. 

1. var headerEntry = rec.headers.find(h => h.key=="MyHeader");
2. var myHeader = String.fromCharCode(...JSON.parse(headerEntry.value))
3. rec.value = myHeader; // Now you can filter, e.g. return myHeader === 'myValue';
4. return rec; // or return the header value
   

• Related Articles

• Architecture of the Quick Processor

  The following diagram illustrates the data flow of a data set. The Quick Processor is executed after the decoding process for each individual data set. A Quick Processor can be used for data transformation, but can also act as a filter with complex ...

• Data Protection Policies and Data Masking

  Data protection is a crucial aspect of business operations, especially when dealing with sensitive information. Kadeck Teams Enterprise is equipped with a Data Protection Policy module that helps you define data protection policies through the ...

• Quick Processor Rights Management

  To modify or create a Quick Processor, the user must have both the QuickProcessorModify right and the TopicAccessRead right. QuickProcessorModify can be assigned for individual connections/environments. In combination with the TopicAccessRead right, ...

• Custom Codecs Introduction

  This article gives a brief introduction of custom codecs in Kadeck Desktop and Kadeck Teams and covers the essential steps to get you up and running quickly. It is recommended to read Structure of Codecs before starting developing your own codec as ...

• Introduction to Monitoring & Health

  We are excited to introduce Kadeck's new Monitoring Module, which makes monitoring and troubleshooting Apache Kafka infrastructure easier than ever and raises the bar in the industry. The new monitoring module includes four new pages, Monitoring ...