Tuesday, June 22, 2021

gRPC and OSGi Remote Services

 gRPC is a popular framework for creating high-performance remote procedure call-based microservices.  

OSGi Remote Services is a transport-agnostic specification for creating dynamic, versionable, modular, remote services.

The ECF project provides an open implementation of the OSGi Remote Services spec, and has a provider implementation based-upon gRPC.   What this means is that gRPC can be used to create and run as an OSGi remote service, with all the support for service dynamics (particularly important for network-based services), versioning, and other features provided by OSGi remote services.

The architectural fit between gRPC and OSGi Remote Services is very good, since gRPC is concerned with transport-level efficiency (i.e. http/2, binary serialization format), and OSGi Remote Services are completely transport-agnostic, and focuses instead upon service-level concerns (e.g. dynamics, versioning, and service discovery).

gRPC offers support for server and client-based streaming.   In ECF's implementation, streaming rpcs are mapped to the reactivex api.  This means that consumers and implementers of a streaming rpc can simply call methods and provide callbacks (using Flowable), and non-blocking streaming calls will be made.  In addition, the use of reactivex and backpressure will result in transport-level flow control for these streaming APIs!

Another advantage of gRPC for OSGi remote services is it's polyglot nature.   This means that if (for example) a gRPC remote service is run as an OSGi/Java server, clients can be easily implemented in any of the languages supported by gRPC.  As well, servers written in some other language can easily created and accessed from OSGi consumers.  An example of this is the ECF etcd3 discovery provider, which communicates with an etcd server (written in Go) to publish and discover OSGi remote services.

Finally, with bndtools (an Eclipse plugin for OSGi bundle development), ECF Remote Service workspace template, and it's support for generating code as part of Eclipse's incremental build, gRPC code generation can be seemlessly integrated into the Eclipse development environment so that gRPC code generation, compile, and bundle packaging can happen immediately and continuously as part of gRPC remote service development.  For a video tutorial demonstrating this, please see here.

Saturday, February 06, 2021

gRPC Code Generation using Bndtools and ECF Remote Services

There's a new video tutorial that demonstrates using ECF Remote Services, bndtools, and Eclipse to create an OSGi Remote Service.   

Bndtools has recently added the ability to run code generators as part of a bnd-based project, and with ECF's bndtools workspace template, a single proto3 file added to a project will automatically generate an entire Java remote service API and update/regenerate the API as changes are made to the proto3 file.   No command-line execution of protoc needed.

Further with ECF's project templates, the generated API can be easily implemented and exported as an OSGi Remote Service.

Please watch the video here

Thursday, January 07, 2021

ECF 3.14.19 released - simplify remote service discovery via properties

 ECF 3.14.19 has been released.

Along with the usual bug fixes, this release includes new documentation on the use of properties for discovering and importing remote services.   The docs describe the use of properties files for simplifying the import of remote services.   

This capability is especially useful for Eclipse RCP clients accessing Jax-RS/REST remote services.

Patrick Paulin describes a production usage his blog posting here.

Tuesday, December 08, 2020

Using properties to simplify discovery of OSGi Remote Services

OSGi Remote Services are discovered by ECF's Remote Services implementation in two ways:  

1. Via a network discovery protocol provider such as:  Zeroconf, jSLP, etcd, Zookeeper, or some custom protocol

2. Via an xml format known as an Endpoint Description Extender Format (EDEF)

 The EDEF format is specified by the OSGi Remote Service Admin specification.   

When importing an EDEF-defined remote service, it's typically necessary to construct the entire EDEF file 'by hand' rather than having he EDEF generated automatically.  This can be quite complicated to construct by hand as some properties are required, others are optional and it's not obvious what all of the values must be for successful import.

A new capability has been added to ECF's Remote Service Admin implementation that allows EDEF Properties to be used with the EDEF, thus simplifying the creation of remote service consumers that use EDEF for import.

This capability was added to support the usage of JaxRS Remote Services in an Eclipse RCP client.  See a description of this use case here.

Monday, September 21, 2020

Using gRPC-java code generation to create OSGi Services

OSGi Services are usually first created by declaring a java service interface class.  As an OSGi service, this interface class serves as both the name for the service in the service registry, and defines the service contract (i.e. the interface method signatures...i.e. the method name, argument types, and return types) for that version of the service.

gRPC (Google RPC) is a popular and high-performance rpc approach that allows developers to define networked services based upon protocol buffers (proto3).

By extending bndtools recently-added code generation capability, it's now possible to generate an OSGi (remote) service API from just a proto3 service declaration.  All the classes necessary for an OSGi Remote Service API (service interface, arg and return types) can be generated by bndtools within Eclipse from a single proto3 file, immediately and completely.

Ready to implement-and-consume OSGi Services can be generated by Eclipse+bndtools + a proto3 service declaration.

Further, the proto3 service declaration can be modified, and the tooling will immediately generate new service API classes, compile, and package them into a bundle, all from within Eclipse+bndtools.

To get this bndtools-grpc generation with an example see here.

Tuesday, July 14, 2020

ECF 3.14.12 released - Now with gRPC for OSGi Remote Services

ECF 3.14.12 was just released

Highlights of this Release

New OSGi Remote Services Distribution provider based upon gRPC/Protocol Buffers. Along with a grpc-osgi-generator project...which allows the generation of a service API from a proto3 service declaration...this provider allows gRPC-based services to be exported and imported as OSGi Remote Services.  This now includes support for unary, server-streaming, and client-streaming gRPC calls.

Enhanced Support for Bndtools-based development of OSGi Remote Services.   The ECF Bndtools Workspace now includes the latest version of ECF Remote Services, along with the gRPC distribution provider, a Hazelcast-based discovery and distribution provider, and project and bndrun templates for creating, running, testing, and debugging OSGi Remote Services in Eclipse+Bndtools 5.

Tuesday, May 05, 2020

Using Google's grpc-java for OSGi Remote Services

A cool thing about Google's grpc is that a service creator can declare a service via a protocol buffers file (.proto file), and then the protoc compiler (along with grpc-java compiler plugin) generates many of the Java classes for both implementing and using that service.

OSGi Remote Services require a service interface to represent the service contract, and this service interface is usually created directly by the programmer.   Through a additional plugin, protoc can now generate a OSGi service interface along with all grpc classes... from the .proto file service declaration.   

For example, consider the following protocol buffers input file:
syntax = "proto3";
package grpc.health.v1;
option java_multiple_files = true;
option java_outer_classname = "HealthProto";
option java_package = "io.grpc.health.v1";
message HealthCheckRequest {
  string message = 1;
message HealthCheckResponse {
  enum ServingStatus {
    UNKNOWN = 0;
    SERVING = 1;
    NOT_SERVING = 2;
    SERVICE_UNKNOWN = 3;  // Used only by the Watch method.
  ServingStatus status = 1;
service HealthCheck {
  // Unary method
  rpc Check(HealthCheckRequest) returns (HealthCheckResponse);
  // Streaming method
  rpc Watch(HealthCheckRequest) returns (stream HealthCheckResponse);
Running protoc+grpc-java+grpc-osgi-generator on this file results in generation of a HealthCheckService class along with all message classes (e.g. HealthCheckRequest, HealthCheckResponse, HealthProto, etc).   All of the Java classes in this example directory were created simply by running protoc+grpc-java+grpc-osgi-generator on the above proto file.

The generated Java classes can then be used to implement an OSGi Remote Service, with HealthCheckService as the service interface.   Tt runtime, the HealthCheckServiceImpl can be exported (via the Grpc Provider) which uses grpc to provide the comm and json serialization for the HealthCheckService method calls.

The net effect is that remote service programmers can easily and quickly go from abstract service declaration (in proto file) to a running/functioning OSGi remote service:
  1. Declare a service in proto file -- example proto file
  2. Run protoc+grpc-java+grpc-osgi-generator to generate the Java code for the declared service - example Java generated code
  3. Implement the service API - example service implementation
  4. Use Declarative Services to export using ECF Remote Services + Grpc Distribution Provider - example (see @Component annotation for OSGi Remote Services-required service properties to trigger export)
The remote service programmer writes no communication nor serialization code (both are provided by the Grpc distribution provider).   See here for the complete generated healthcheck api plugin, here for the impl plugin, and here for a simple remote service consumer.