New Release: Python<->Java Remote Services

There is a new release (2.9.0) of the ECF distribution provider for OSGi R7 Remote Services between Java and Python.

This release has:

An upgraded version of Py4j
An upgraded version of Google Protocol Buffers
Enhancements to the distribution provider based upon the improved Py4j and Protobuf libs

In this previous blog posting there are links to tutorials and examples showing how to use remote services between Python<->Java.

Python<->Java remote services can be consumed or implemented in either Java or Python.

OSGi Remote Services Between Python and Java

In a previous post, I described the support for OSGi Remote Services and Remote Service Admin in iPOPO 0.8.0 release.   The previous post refers to a tutorial showing a Python service impl and Python consumer.

Python<->Java Distribution Provider

Included with iPOPO 0.8.0 is a distribution provider that allows remote services between Java and Python frameworks.   For example, this is a tutorial, that uses Karaf on the Java side as the remote service implementation, and has a Python consumer that calls the Java-implemented remote service.

Python Service Implementation with Java Consumers

This distribution provider also supports Python-implemented remote services, with Java/OSGi consumers.   With iPOPO for dynamic service injection in Python, and Declarative Services for Java/OSGi, this allows very easy Python<->Java service-level interaction, with support for all dynamics, RSA management agent, extensible/customizable topology management, management of complicated service dependencies, pluggable local and network discovery, and other RS/RSA features handled consistently in both Java and Python.  As well, the use of OSGi-specified EndpointDescription service metadata allows service-level interoperability across languages.

Python<->Java with Protocol Buffers Serialization

Also included with this distribution provider is serialization using Google's protocol buffers.  This allows open, extensible, and efficient rpc between Python and Java.

ECF Photon supports OSGI R7

ECF Photon has several major enhancements.   I'll blog about these enhancements individually over the coming weeks, starting with

Support for OSGI R7 Remote Services

In the R7 final draft specification (chapter 100) detail was added about the use of Remote Service Intents.   RS Intents describe a distribution provider's abstract capabilities.    By way of example, several new standard intents have been defined, including osgi.basic and osgi.async.   

The osgi.basic intent requires that a distribution provider support a remote service-specific timeout, as well as serialization of remote service arguments and return values include DTOs (Data Type Objects), java primitives, maps, collections, lists, OSGI Version, etc.   

The osgi.async intent requires that remote service method signatures using CompletableFuture, Future, and OSGI's Promise be supported directly by the distribution provider.   This allows non-blocking asynchronous remote services to be easily declared in the service interface, and implemented by the distribution provider.  Here is an example remote service declaration that demonstrates how the osgi.async intent can be used.   In a forthcoming tutorial, I'll show how such a small service can be easily defined, implemented, and remoted using ECF Remote Services.

ECF's remote services impl has multiple distribution providers, and open APIs for creating custom or extension providers.   Most of the existing ECF distribution providers already available have been updated to implement the R7-standardized intents.   Others will be updated prior to and after Photon release.

Python 3 and Import Hooks for OSGi Services

In a previous post I described using Python for implementing OSGi Services.   This Python<->Java service bridge allows Python-provided/implemented OSGi services called from Java, and Java-provided/implemented OSGi Services called from Python.   OSGi Remote Services provides a standardized way of communicating service meta-data (e.g. service contracts, endpoint meta-data) between Java and Python processes.

As this Java<->Python communication conforms to the OSGi Remote Services specification, everything is completely inter-operable with Declarative Services and/or other frameworks based upon OSGi Services.  It will also run in any OSGi R5+ environment, including Eclipse, Karaf, OSGi-based web servers, or other OSGi-based environments.

Recently, Python 3 has introduced the concept of an Import Hook.   An import hook allows the python path and the behavior of the python import statement to be dynamically or extended. 

In the most recent version (2.7) of the ECF Py4j Distribution Provider, we use import hooks so that Python module import is resolved by a Java-side OSGi ModuleResolver service.   For example, as described in this tutorial, this Python statement

from hello import HelloServiceImpl

imports the hello.py module as a string loaded from within an OSGi bundle.  Among other things, this allows OSGi dynamics to be used to add and remove modules from the python path without stopping and restarting either the Java or the Python processes.

Remote Services between Python and Java

ECF's implementation of OSGi Remote Services allows multiple distribution providers, which are responsible for the actual rpc communication required by remote services.   Here is a list of ECF distribution providers we've created.

Using Py4j and Google Protocol Buffers, we've recently enhanced an ECF distribution provider that allows the use of remote services (and Remote Service Admin) between OSGi and Python.   Service impls can be in either Java or Python, and consumers can be either Java or Python.     Protocol Buffers can be used to efficiently serialize arguments and return values.

The only dependencies are on OSGi, Py4j, and Google Protocol buffers, so this distribution provider can be used in Eclipse or other OSGi environments like Karaf.

Get the most recent release, with examples and source code at this github repository.

Microservices Granularity for the Internet of Things

In a 2014 blog posting, Martin Fowler discussed issues around creating networked services in Microservices and the First Law of Distributed Objects.   One of his points is that networked services should generally be more coarse-grained than local (in-process) services. The reason for this is that distribution always has costs (bandwidth, performance), and these costs easily can become large with fine-grained remote calls.

But as Fowler points out, there are good reasons (e.g. complexity) to make a networked API as fine-grained as possible.   How granular/coarse should IoT microservices be?  In his blog posting, Fowler suggested that granularity was an open question, and that experience with different systems with different levels of microservices granularity would provide eventual insight.

I agree with Fowler's view that experience is necessary to decide on 'appropriate' granularity.   I think it's particularly true for the Internet of Things, where multiple people and organizations are attempting attempting to create consistent abstractions for the relatively-limited input and output capabilities exposed by newly networked devices...aka 'things'.

But when actually defining remote services, often the first thing done is to bind the service to a particular transport+protocol+impl framework (e.g. https+json+jersey).   Once bound to a transport, the service API may become very difficult to refactor and version.  This is especially true once a service has been deployed, but frequently deployment is the only way to get enough real experience to be more (or less) granular!

One way to provide flexibility...and allow future change to a service is to remain as transport-independent as possible.   As described by this article, new standards such as OSGi Remote Services/RSA and ECF's modular implementation makes it possible to design and refactor services independent of the transport.  Such independence will make it easier to update the granularity of a microservice when necessary.