Administration and Architectural Enhancements

The following data administration and architectural enhancements were made in Zoomdata 4:

  1. Load balancing for the query engine was introduced. Several instances of the query engine can be registered simultaneously and Zoomdata can distribute queries among them in a round-robin fashion. This ensures that you get uninterrupted service, even when a query engine instance goes down.

    When a user session or request is being processed by a query engine instance and the instance fails or is stopped intentionally, Zoomdata automatically switches the user session or request to a healthy query engine instance with minimal disruption. If no query engine instance is available, the user is notified and work resumes automatically as soon as a query engine instance becomes available. The query engine instances are interchangeable.

  2. Connector load balancing was introduced. Using a new Consul service registration model, several instances of the same connector can be registered simultaneously and Zoomdata can distribute queries among them in a round-robin fashion. This ensures that you get uninterrupted service, even when a connector instance goes down. The unique connector server URLs for started connector instances appear in the list of connector servers on the Manage Connector Services page.

    All Zoomdata 4 connectors use this new service registration model. However, the new model is not fully compatible with earlier versions of Zoomdata. Special configuration steps must be performed if you want to use a Zoomdata version 4 connector with an earlier version of Zoomdata or use a Zoomdata version 3.1 (or earlier) connector with Zoomdata 4 (or later).

    Zoomdata 3.2 to 3.7 connectors are fully compatible with Zoomdata 4.

    See Migrating Connectors for Other Zoomdata Versions for more information.

  3. This release introduces support for high availability of your Zoomdata environment. In a high availability scenario you can minimize the downtime caused by:

    • hardware or software failures due to excessive resource use or other events
    • software upgrades or updates

    In a high availability Zoomdata environment, you can deploy multiple instances of Zoomdata to ensure that at least one instance of its microservices operates continuously. A load balancer is required to distribute the network traffic across your user-facing Zoomdata nodes. Microservice load balancing and failover occur automatically within the Zoomdata nodes themselves. In addition, you can monitor all microservices and collect diagnostic trace information for them using the Service Monitor in conjunction with the Tracing microservice. For more information, see High Availability Environment Support and Configure a Zoomdata v4 High Availability Environment.

    If you have the configuration microservice configured and running, you can maintain properties for microservices of a given type in a single location in the Service Monitor. For example, if you have two query engine microservices running in your high availability environment, you can change the properties for both microservices in a single location, ensuring that the query engine microservices operate in the same manner across the product nodes. A new config-server-upload.jar utility is provided that can be used to migrate the microservice properties from your standalone Zoomdata servers to the Zoomdata configuration data in the high availability PostgreSQL data store, where the configuration microservice can maintain them. For more information, see Migrating Properties to the Configuration Server.

  4. With this release, all Zoomdata queries are processed using the enhanced query engine, called zEngine that we have supported for fused data queries since Zoomdata 3.6. Note the following considerations when using zEngine:

    1. Data Sharpening is not currently supported with zEngine. If your organization requires Data Sharpening in your data analytics use cases, you will need to reenable the legacy query engine. See Reenabling the Legacy Query Engine.

      When you reenable the legacy query engine, data fusion will no longer function because it requires zEngine.
    2. Several major changes in calculating the previous period (using the PreviousPeriod function) take effect when using zEngine:

      • Specifying PreviousPeriod() is no longer allowed. Parameters are now required.

      • If the data is grouped by the same field for which a PreviousPeriod transformation is performed and it is grouped by days but transformed by units of months, quarters, or years, zEngine returns null values when the previous period does not have matching days for the current period.

        For example, if the current period is the month of March 2015 and PreviousPeriod('months',1) is used for the transformation, zEngine produces null values for February 29-31, 2015, because those days are not valid days (although they are valid days for March 2015).

        The zEngine attempts to preserve the day-of-month correspondence between the two periods (the previous query engine did not consistently do this).

        See PreviousPeriod Function.

      • Playback and live mode are now supported when even time intervals are requested. See Live Mode and Historical Playback and Even Time Intervals.

      • The totals in pivot tables that use aggregate (group) filters no longer display as Null.

    For complete information about the query engine, see Query Engine Microservice.

  5. Graceful shutdown of the query engine was introduced. When the query engine is shut down, it gracefully completes queries that are in-flight and notifies clients that it is terminating. Several new query engine properties have been introduced to support graceful shutdown. See Query Engine Graceful Shutdown.

  6. The Zoomdata Streamwriter microservice was optimized and renamed in this release. It is now called the Data Writer microservice.

    • Its new microservice name is zoomdata-data-writer-postgresql.service.
    • The new properties file is called
    • All of the original Streamwriter microservice properties have been renamed with data-writer replacing streamwriter in their property names. The following Data Writer properties must be set:

      Property Description Default
      data-writer.service.url The Data Writer microservice URL http://localhost:8081/‌ The Data Writer microservice name data-writer

    As a result of the Data Writer optimization changes, the use of RabbitMQ and the Zoomdata upload microservice are no longer needed and are deprecated.

  7. The use of the Zoomdata Scheduler microservice has been deprecated in this release. Because of this change, migration steps must be performed to upgrade to Zoomdata 4. See Zoomdata Scheduler Migration.

  8. This version introduces the ability to manage the application properties of Zoomdata's dependent microservices using the Zoomdata Service Monitor. This provides a centralized location where Zoomdata configuration properties can be maintained. However, it requires that the new Zoomdata configuration microservice first be configured and started. See Using the Zoomdata Configuration Microservice to Maintain Application Properties.

  9. If you click on the URL for an instance of any of the following microservices listed on the Service Monitor's Applications View page, a page of metrics and other information now appears for that instance of the microservice.

    • all data connector microservices

    • Screenshot microservice

    • Data Writer microservice

    • query engine microservice

    • Service Discovery microservice

    • tracing microservice

    • configuration microservice

    See Applications View and Zoomdata Microservice Name Reference.