Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .idea/jpa-buddy.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions .idea/logstash-docs.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/misc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

85 changes: 85 additions & 0 deletions .idea/workspace.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions docs/versioned-plugins/codecs/avro-index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ include::{include_path}/version-list-intro.asciidoc[]

|=======================================================================
| Version | Release Date
| <<v3.4.1-plugins-codecs-avro,v3.4.1>> | 2023-10-16
| <<v3.4.0-plugins-codecs-avro,v3.4.0>> | 2022-04-28
| <<v3.3.1-plugins-codecs-avro,v3.3.1>> | 2021-12-07
| <<v3.3.0-plugins-codecs-avro,v3.3.0>> | 2021-08-19
Expand All @@ -15,6 +16,7 @@ include::{include_path}/version-list-intro.asciidoc[]
| <<v3.2.0-plugins-codecs-avro,v3.2.0>> | 2017-05-19
|=======================================================================

include::avro-v3.4.1.asciidoc[]
include::avro-v3.4.0.asciidoc[]
include::avro-v3.3.1.asciidoc[]
include::avro-v3.3.0.asciidoc[]
Expand Down
158 changes: 158 additions & 0 deletions docs/versioned-plugins/codecs/avro-v3.4.1.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
:plugin: avro
:type: codec

///////////////////////////////////////////
START - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////
:version: v3.4.1
:release_date: 2023-10-16
:changelog_url: https://github.com/logstash-plugins/logstash-codec-avro/blob/v3.4.1/CHANGELOG.md
:include_path: ../include/6.x
///////////////////////////////////////////
END - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////

[id="{version}-plugins-{type}s-{plugin}"]

=== Avro codec plugin {version}

include::{include_path}/plugin_header.asciidoc[]

==== Description

Read serialized Avro records as Logstash events

This plugin is used to serialize Logstash events as
Avro datums, as well as deserializing Avro datums into
Logstash events.

[id="{version}-plugins-{type}s-{plugin}-ecs_metadata"]
==== Event Metadata and the Elastic Common Schema (ECS)

The plugin behaves the same regardless of ECS compatibility, except adding the original message to `[event][original]`.

==== Encoding

This codec is for serializing individual Logstash events
as Avro datums that are Avro binary blobs. It does not encode
Logstash events into an Avro file.


==== Decoding

This codec is for deserializing individual Avro records. It is not for reading
Avro files. Avro files have a unique format that must be handled upon input.

.Partial deserialization
[NOTE]
================================================================================
Avro format is known to support partial deserialization of arbitrary fields,
providing a schema containing a subset of the schema which was used to serialize
the data.
This codec *doesn't support partial deserialization of arbitrary fields*.
Partial deserialization _might_ work only when providing a schema which contains
the first `N` fields of the schema used to serialize the data (and
in the same order).
================================================================================

==== Usage
Example usage with Kafka input.

[source,ruby]
----------------------------------
input {
kafka {
codec => avro {
schema_uri => "/tmp/schema.avsc"
}
}
}
filter {
...
}
output {
...
}
----------------------------------

[id="{version}-plugins-{type}s-{plugin}-options"]
==== Avro Codec Configuration Options

[cols="<,<,<",options="header",]
|=======================================================================
|Setting |Input type|Required
| <<{version}-plugins-{type}s-{plugin}-ecs_compatibility>> | {logstash-ref}/configuration-file-structure.html#string[string]|No
| <<{version}-plugins-{type}s-{plugin}-encoding>> | {logstash-ref}/configuration-file-structure.html#string[string], one of `["binary", "base64"]`|No
| <<{version}-plugins-{type}s-{plugin}-schema_uri>> |{logstash-ref}/configuration-file-structure.html#string[string]|Yes
| <<{version}-plugins-{type}s-{plugin}-tag_on_failure>> |{logstash-ref}/configuration-file-structure.html#boolean[boolean]|No
| <<{version}-plugins-{type}s-{plugin}-target>> |{logstash-ref}/configuration-file-structure.html#string[string]|No
|=======================================================================

&nbsp;

[id="{version}-plugins-{type}s-{plugin}-ecs_compatibility"]
===== `ecs_compatibility`

* Value type is {logstash-ref}/configuration-file-structure.html#string[string]
* Supported values are:
** `disabled`: Avro data added at root level
** `v1`,`v8`: Elastic Common Schema compliant behavior (`[event][original]` is also added)

Controls this plugin's compatibility with the {ecs-ref}[Elastic Common Schema (ECS)].

[id="{version}-plugins-{type}s-{plugin}-encoding"]
===== `encoding`

* Value can be any of: `binary`, `base64`
* Default value is `base64`

Set encoding for Avro's payload.
Use `base64` (default) to indicate that this codec sends or expects to receive base64-encoded bytes.

Set this option to `binary` to indicate that this codec sends or expects to receive binary Avro data.


[id="{version}-plugins-{type}s-{plugin}-schema_uri"]
===== `schema_uri`

* This is a required setting.
* Value type is {logstash-ref}/configuration-file-structure.html#string[string]
* There is no default value for this setting.

schema path to fetch the schema from.
This can be a 'http' or 'file' scheme URI
example:

* http - `http://example.com/schema.avsc`
* file - `/path/to/schema.avsc`

[id="{version}-plugins-{type}s-{plugin}-tag_on_failure"]
===== `tag_on_failure`

* Value type is {logstash-ref}/configuration-file-structure.html#boolean[boolean]
* Default value is `false`

tag events with `_avroparsefailure` when decode fails

[id="{version}-plugins-{type}s-{plugin}-target"]
===== `target`

* Value type is {logstash-ref}/configuration-file-structure.html#string[string]
* There is no default value for this setting.
* This is only relevant when decode data into an event

Define the target field for placing the values. If this setting is not
set, the Avro data will be stored at the root (top level) of the event.

*Example*
[source,ruby]
----------------------------------
input {
kafka {
codec => avro {
schema_uri => "/tmp/schema.avsc"
target => "[document]"
}
}
}
----------------------------------
2 changes: 2 additions & 0 deletions docs/versioned-plugins/codecs/protobuf-index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ include::{include_path}/version-list-intro.asciidoc[]

|=======================================================================
| Version | Release Date
| <<v1.3.0-plugins-codecs-protobuf,v1.3.0>> | 2023-09-20
| <<v1.2.9-plugins-codecs-protobuf,v1.2.9>> | 2023-04-11
| <<v1.2.8-plugins-codecs-protobuf,v1.2.8>> | 2023-03-28
| <<v1.2.5-plugins-codecs-protobuf,v1.2.5>> | 2021-02-08
Expand All @@ -16,6 +17,7 @@ include::{include_path}/version-list-intro.asciidoc[]
| <<v1.0.2-plugins-codecs-protobuf,v1.0.2>> | 2017-06-23
|=======================================================================

include::protobuf-v1.3.0.asciidoc[]
include::protobuf-v1.2.9.asciidoc[]
include::protobuf-v1.2.8.asciidoc[]
include::protobuf-v1.2.5.asciidoc[]
Expand Down
Loading
Loading