Skip to content

Commit 78b8386

Browse files
sleipnirpolvalente
sleipnir
and
polvalente
authored
[Feat]: Add support for streams pipelines (#418)
* feat: added stream pipeline support * feat: add streams app example * chore: added reflection to hello streams example for easy testing * feat: added new factory function * feat: added support for header propagation * feat: added support for header propagation * chore: adjusts * feat: added new replace operator * new line * refactor: propagate context option * refactor: rename functions * refactor: removing the destructive version * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * adjusts * refactor: separating unary from non-unary * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * chore: remove reject function * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * chore: format * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * refactor: rename operator reject to via * chore: pointing flow docs * fix: correct api examples * chore: change ask and join_with apis * refactor: rename single to unary * refactor: rename single to unary in tests * chore: remove unascessary function * chore: removes a not so necessary function * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * Update lib/grpc/stream.ex Co-authored-by: Paulo Valente <[email protected]> * fix: format --------- Co-authored-by: Adriano Santos <[email protected]> Co-authored-by: Paulo Valente <[email protected]>
1 parent 917f3ff commit 78b8386

File tree

20 files changed

+1127
-4
lines changed

20 files changed

+1127
-4
lines changed

‎examples/helloworld_streams/.formatter.exs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Used by "mix format"
2+
[
3+
inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"]
4+
]

‎examples/helloworld_streams/.gitignore‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# The directory Mix will write compiled artifacts to.
2+
/_build/
3+
4+
# If you run "mix test --cover", coverage assets end up here.
5+
/cover/
6+
7+
# The directory Mix downloads your dependencies sources to.
8+
/deps/
9+
10+
# Where third-party dependencies like ExDoc output generated docs.
11+
/doc/
12+
13+
# Ignore .fetch files in case you like to edit your project deps locally.
14+
/.fetch
15+
16+
# If the VM crashes, it generates a dump, let's ignore it too.
17+
erl_crash.dump
18+
19+
# Also ignore archive artifacts (built via "mix archive.build").
20+
*.ez
21+
22+
# Ignore package tarball (built via "mix hex.build").
23+
helloworld_streams-*.tar
24+
25+
# Temporary files, for example, from tests.
26+
/tmp/

‎examples/helloworld_streams/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# HelloworldStreams
2+
3+
**TODO: Add description**
4+
5+
## Installation
6+
7+
If [available in Hex](https://hex.pm/docs/publish), the package can be installed
8+
by adding `helloworld_streams` to your list of dependencies in `mix.exs`:
9+
10+
```elixir
11+
def deps do
12+
[
13+
{:helloworld_streams, "~> 0.1.0"}
14+
]
15+
end
16+
```
17+
18+
Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)
19+
and published on [HexDocs](https://hexdocs.pm). Once published, the docs can
20+
be found at <https://hexdocs.pm/helloworld_streams>.
21+

‎examples/helloworld_streams/lib/helloworld_streams.ex‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
defmodule HelloworldStreams do
2+
@moduledoc """
3+
Documentation for `HelloworldStreams`.
4+
"""
5+
end

‎examples/helloworld_streams/lib/helloworld_streams/application.ex‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
defmodule HelloworldStreams.Application do
2+
@moduledoc false
3+
use Application
4+
5+
@impl true
6+
def start(_type, _args) do
7+
children = [
8+
HelloworldStreams.Utils.Transformer,
9+
GrpcReflection,
10+
{
11+
GRPC.Server.Supervisor,
12+
endpoint: HelloworldStreams.Endpoint, port: 50053, start_server: true
13+
}
14+
]
15+
16+
opts = [strategy: :one_for_one, name: HelloworldStreams.Supervisor]
17+
Supervisor.start_link(children, opts)
18+
end
19+
end

‎examples/helloworld_streams/lib/helloworld_streams/endpoint.ex‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
defmodule HelloworldStreams.Endpoint do
2+
@moduledoc false
3+
use GRPC.Endpoint
4+
5+
intercept(GRPC.Server.Interceptors.Logger)
6+
run(HelloworldStreams.Utils.Reflection)
7+
run(HelloworldStreams.Server)
8+
end

‎examples/helloworld_streams/lib/helloworld_streams/server.ex‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
defmodule HelloworldStreams.Server do
2+
@moduledoc """
3+
gRPC service for streaming data.
4+
"""
5+
use GRPC.Server, service: Stream.EchoServer.Service
6+
7+
alias HelloworldStreams.Utils.Transformer
8+
alias GRPC.Stream, as: GRPCStream
9+
10+
alias Stream.HelloRequest
11+
alias Stream.HelloReply
12+
13+
@spec say_unary_hello(HelloRequest.t(), GRPC.Server.Stream.t()) :: any()
14+
def say_unary_hello(request, _materializer) do
15+
GRPCStream.unary(request)
16+
|> GRPCStream.ask(Transformer)
17+
|> GRPCStream.map(fn %HelloReply{} = reply ->
18+
%HelloReply{message: "[Reply] #{reply.message}"}
19+
end)
20+
|> GRPCStream.run()
21+
end
22+
23+
@spec say_server_hello(HelloRequest.t(), GRPC.Server.Stream.t()) :: any()
24+
def say_server_hello(request, materializer) do
25+
create_output_stream(request)
26+
|> GRPCStream.from()
27+
|> GRPCStream.run_with(materializer)
28+
end
29+
30+
defp create_output_stream(msg) do
31+
Stream.repeatedly(fn ->
32+
index = :rand.uniform(10)
33+
%HelloReply{message: "[#{index}] I'm the Server for #{msg.name}"}
34+
end)
35+
|> Stream.take(10)
36+
|> Enum.to_list()
37+
end
38+
39+
@spec say_bid_stream_hello(Enumerable.t(), GRPC.Server.Stream.t()) :: any()
40+
def say_bid_stream_hello(request, materializer) do
41+
# simulate a infinite stream of data
42+
# this is a simple example, in a real world application
43+
# you would probably use a GenStage or similar
44+
# to handle the stream of data
45+
output_stream =
46+
Stream.repeatedly(fn ->
47+
index = :rand.uniform(10)
48+
%HelloReply{message: "[#{index}] I'm the Server ;)"}
49+
end)
50+
51+
GRPCStream.from(request, join_with: output_stream)
52+
|> GRPCStream.map(fn
53+
%HelloRequest{} = hello ->
54+
%HelloReply{message: "Welcome #{hello.name}"}
55+
56+
output_item ->
57+
output_item
58+
end)
59+
|> GRPCStream.run_with(materializer)
60+
end
61+
end

‎examples/helloworld_streams/lib/helloworld_streams/stream.pb.ex‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
defmodule Stream.HelloRequest do
2+
@moduledoc false
3+
4+
use Protobuf, protoc_gen_elixir_version: "0.14.0", syntax: :proto3
5+
6+
def descriptor do
7+
# credo:disable-for-next-line
8+
%Google.Protobuf.DescriptorProto{
9+
name: "HelloRequest",
10+
field: [
11+
%Google.Protobuf.FieldDescriptorProto{
12+
name: "name",
13+
extendee: nil,
14+
number: 1,
15+
label: :LABEL_OPTIONAL,
16+
type: :TYPE_STRING,
17+
type_name: nil,
18+
default_value: nil,
19+
options: nil,
20+
oneof_index: nil,
21+
json_name: "name",
22+
proto3_optional: nil,
23+
__unknown_fields__: []
24+
}
25+
],
26+
nested_type: [],
27+
enum_type: [],
28+
extension_range: [],
29+
extension: [],
30+
options: nil,
31+
oneof_decl: [],
32+
reserved_range: [],
33+
reserved_name: [],
34+
__unknown_fields__: []
35+
}
36+
end
37+
38+
field(:name, 1, type: :string)
39+
end
40+
41+
defmodule Stream.HelloReply do
42+
@moduledoc false
43+
44+
use Protobuf, protoc_gen_elixir_version: "0.14.0", syntax: :proto3
45+
46+
def descriptor do
47+
# credo:disable-for-next-line
48+
%Google.Protobuf.DescriptorProto{
49+
name: "HelloReply",
50+
field: [
51+
%Google.Protobuf.FieldDescriptorProto{
52+
name: "message",
53+
extendee: nil,
54+
number: 1,
55+
label: :LABEL_OPTIONAL,
56+
type: :TYPE_STRING,
57+
type_name: nil,
58+
default_value: nil,
59+
options: nil,
60+
oneof_index: nil,
61+
json_name: "message",
62+
proto3_optional: nil,
63+
__unknown_fields__: []
64+
}
65+
],
66+
nested_type: [],
67+
enum_type: [],
68+
extension_range: [],
69+
extension: [],
70+
options: nil,
71+
oneof_decl: [],
72+
reserved_range: [],
73+
reserved_name: [],
74+
__unknown_fields__: []
75+
}
76+
end
77+
78+
field(:message, 1, type: :string)
79+
end
80+
81+
defmodule Stream.EchoServer.Service do
82+
@moduledoc false
83+
84+
use GRPC.Service, name: "stream.EchoServer", protoc_gen_elixir_version: "0.14.0"
85+
86+
def descriptor do
87+
# credo:disable-for-next-line
88+
%Google.Protobuf.ServiceDescriptorProto{
89+
name: "EchoServer",
90+
method: [
91+
%Google.Protobuf.MethodDescriptorProto{
92+
name: "SayUnaryHello",
93+
input_type: ".stream.HelloRequest",
94+
output_type: ".stream.HelloReply",
95+
options: %Google.Protobuf.MethodOptions{
96+
deprecated: false,
97+
idempotency_level: :IDEMPOTENCY_UNKNOWN,
98+
features: nil,
99+
uninterpreted_option: [],
100+
__pb_extensions__: %{},
101+
__unknown_fields__: []
102+
},
103+
client_streaming: false,
104+
server_streaming: false,
105+
__unknown_fields__: []
106+
},
107+
%Google.Protobuf.MethodDescriptorProto{
108+
name: "SayServerHello",
109+
input_type: ".stream.HelloRequest",
110+
output_type: ".stream.HelloReply",
111+
options: %Google.Protobuf.MethodOptions{
112+
deprecated: false,
113+
idempotency_level: :IDEMPOTENCY_UNKNOWN,
114+
features: nil,
115+
uninterpreted_option: [],
116+
__pb_extensions__: %{},
117+
__unknown_fields__: []
118+
},
119+
client_streaming: false,
120+
server_streaming: true,
121+
__unknown_fields__: []
122+
},
123+
%Google.Protobuf.MethodDescriptorProto{
124+
name: "SayBidStreamHello",
125+
input_type: ".stream.HelloRequest",
126+
output_type: ".stream.HelloReply",
127+
options: %Google.Protobuf.MethodOptions{
128+
deprecated: false,
129+
idempotency_level: :IDEMPOTENCY_UNKNOWN,
130+
features: nil,
131+
uninterpreted_option: [],
132+
__pb_extensions__: %{},
133+
__unknown_fields__: []
134+
},
135+
client_streaming: true,
136+
server_streaming: true,
137+
__unknown_fields__: []
138+
}
139+
],
140+
options: nil,
141+
__unknown_fields__: []
142+
}
143+
end
144+
145+
rpc(:SayUnaryHello, Stream.HelloRequest, Stream.HelloReply)
146+
147+
rpc(:SayServerHello, Stream.HelloRequest, stream(Stream.HelloReply))
148+
149+
rpc(:SayBidStreamHello, stream(Stream.HelloRequest), stream(Stream.HelloReply))
150+
end
151+
152+
defmodule Stream.EchoServer.Stub do
153+
@moduledoc false
154+
155+
use GRPC.Stub, service: Stream.EchoServer.Service
156+
end

‎examples/helloworld_streams/lib/helloworld_streams/utils/reflection.ex‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
defmodule HelloworldStreams.Utils.Reflection do
2+
@moduledoc """
3+
gRPC reflection server.
4+
"""
5+
use GrpcReflection.Server,
6+
version: :v1,
7+
services: [Stream.EchoServer.Service]
8+
end

‎examples/helloworld_streams/lib/helloworld_streams/utils/transformer.ex‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
defmodule HelloworldStreams.Utils.Transformer do
2+
@moduledoc """
3+
`Transformer` GenServer for example purposes.
4+
"""
5+
use GenServer
6+
7+
alias Stream.HelloRequest
8+
alias Stream.HelloReply
9+
10+
def start_link(_) do
11+
GenServer.start_link(__MODULE__, nil, name: __MODULE__)
12+
end
13+
14+
def init(_), do: {:ok, %{}}
15+
16+
def handle_info({:request, %HelloRequest{} = value, from}, state) do
17+
Process.send(from, {:response, %HelloReply{message: "Hello #{value.name}"}}, [])
18+
{:noreply, state}
19+
end
20+
end

0 commit comments

Comments
 (0)