HERE Workspace & Marketplace 2.14 release
Highlights
Customize Volatile layer storage capacity and redundancy to save cost
The configuration options for Volatile layers have been updated to replace "Package Type" with the ability to customize storage capacity in increments of 100 MB. With this new feature, you can create custom sized Volatile layers where the minimum and default size is 100 MB so that you can tailor your storage capacity to your use case and budget requirements. Further, you can choose to store your volatile data redundantly or not. Together, these new options can help you greatly reduce and optimize the money you spend on Volatile layer storage.
Note:
- Customizable Volatile layer storage capacity is possible for new Volatile layers only.
- While selecting no data redundancy does save cost, doing so waives all volatile layer data durability support and should be considered high risk.
- Documentation is updated to clarify Volatile layer storage redundancy options, factors and costs.
- Customers using Data Client Library versions less than 0.6.x will continue to see package types and sizes representative of various configured custom sizes (multiple size configurations can return the same package type response since the package sizes differ greatly).
Access additional HERE Map Content
The HERE Map Content catalog will start publishing a significant amount of additional content. The following catalog layers will be populated:
- Recreational Vehicle Attributes
- Bicycle Attributes
- Complex Road Attributes
- Warning Locations
- Truck Service Locations
- Fueling Stations
- Toll Cost
- Vehicle Regulations
- Flag Images
- Generalized Junctions & Signs
- 2D Landmarks
- 3D Landmarks
All content will initially be offered without an additional premium content uplift for 8 weeks. After that, charges will apply.
Note: Generalized Junctions & Signs, 2D Landmarks, 3D Landmarks are map-reference layers only, so HERE Visuals files can be properly coupled to the HERE Map. The actual product files ( e.g. landmark models ) will be published in a new Workspace catalog called “HERE Map Content – Visuals" in the near future.
Visually inspect all tiled HERE Map Content layers
All tiled HERE Map Content layers have rendering scripts associated with them, that allows inspecting partition payloads visually via the Platform Portal (Inspect function). The scripts implement a basic styling of features, which can be easily overridden. Some initial examples are provided below.
Access additional HERE Map Content: Navigation Attributes via the Optimized Map for Location Library
The following navigation attributes from the HERE Map Content catalog are now available: Speed Limit, Supplemental Geometry, Travel Direction, and Urban flag. We will continue to iteratively compile HERE Map Content attributes into the Optimized Map for Location Library enabling fast and simplified direct access to these attributes via the Location Library. For more detail on the added attributes, please see: HERE Map Content- Navigation Attributes layer
Get service reports in the support portal
Platinum and Gold level support customers now have access to service reports including topics like latency and availability within the support portal.
Changes, Additions and Known Issues
SDK for Java and Scala
To read more information about the SDK for Java and Scala, please visit the HERE Open Location Platform Changelog.
Added: A new SDK component, the Data Client Base Library, is released and provides direct access to Data APIs. Along with this direct access, this library:
- Has no Akka dependencies so you do not need to interact with or understand Akka.
- Provides full control over retry strategies (number of retries, time-outs, etc.)
- Provides control over HTTP headers
- Enables counters and metrics per endpoint
- Is integrated with all Data APIs which were not included in the Data Client Library (e.g. the Ingest API, Notifications, etc.).
This component is separate from the Data Client Library. Developers wishing to have more direct interaction with Data APIs for more specific/custom solutions can leverage this Data Client Base Library. This component opens up Data APIs directly through a standalone client on a programmatic level.
SDK for C++
To read more information about the SDK for C++, please visit the HERE Open Location Platform Changelog.
Added: The option to use local system time for authentication.
Added: The option to prefetch data of Volatile layers.
Added: API to remove a single partition or tilekey from cache for Volatile and Versioned layers.
SDK for Python
Added: Write support for Versioned layers. A one step script has been added to simplify the installation, refer to the SDK for Python Set Up Guide for more details.
Web & Portal
Issue: The custom run-time configuration for a Pipeline Version has a limit of 64 characters for the property name and 255 characters for the value.
Workaround: For the property name, you can define a shorter name in the configuration and map that to the actual, longer name within the pipeline code. For the property value, you must stay within the limitation.
Issue: Pipeline Templates can't be deleted from the Portal UI.
Workaround: Use the CLI or API to delete Pipeline Templates.
Issue: In the Portal, new jobs and operations are not automatically added to the list of jobs and operations for a pipeline version while the list is open for viewing.
Workaround: Refresh the Jobs and Operations pages to see the latest job or operation in the list.
Projects & Access Management
Issue: A finite number of access tokens (~ 250) are available for each app or user. Depending on the number of resources included, this number may be smaller.
Workaround: Create a new app or user if you reach the limitation.
Issue: Only a finite number of permissions are allowed for each app or user in the system across all services. It will be reduced depending on the inclusion of resources and types of permissions.
Issue: All users and apps in a group are granted permissions to perform all actions on any pipeline associated with that group. There is no support for users or apps with limited permissions. For example, you cannot have a reduced role that can only view pipeline status, but not start and stop a pipeline.
Workaround: Limit the users in a pipeline's group to only those users who should have full control over the pipeline.
Issue: When updating permissions, it can take up to an hour for changes to take effect.
Issue: Projects and all resources in a Project are designed for use only in Workspace and are unavailable for use in Marketplace. For example, a catalog created in a Platform Project can only be used in that Project. It cannot be marked as "Marketplace ready" and cannot be listed in the Marketplace.
Workaround: Do not create catalogs in a Project when they are intended for use in both Workspace and Marketplace.
Data
Deprecated: The Volatile layer configuration option to set storage capacity as a "Package Type" will be deprecated within (6) months or by October 30, 2020. All customers should deprecate their existing volatile layers and create new volatile layers with these new configurations within (6) months of this feature release or by October 30, 2020.
Issue: The changes released with 2.9 (RoW) and with 2.10 (China) to add OrgID to Catalog HRNs and with 2.10 (Global) to add OrgID to Schema HRNs could impact any use case (CI/CD or other) where comparisons are performed between HRNs used by various workflow dependencies. For example, requests to compare HRNs that a pipeline is using vs what a Group, User or App has permissions to will result in errors if the comparison is expecting results to match the old HRN construct. With this change, Data APIs will return only the new HRN construct which includes the OrgID (e.g. olp-here…) so a comparison between the old HRN and the new HRN will be unsuccessful.
- Reading from and writing to Catalogs using old HRNs is not broken and will continue to work until July 31, 2020.
- Referencing old Schema HRNs is not broken and will work into perpetuity.
Workaround: Update any workflows comparing HRNs to perform the comparison against the new HRN construct, including OrgID.
Issue: Versions of the Data Client Library prior to 2.9 did not compress or decompress data correctly per configurations set in Stream layers. We changed this behavior in 2.9 to strictly adhere to the compression setting in the Stream layer configuration but when doing so, we broke backward compatibility wherein data ingested and consumed via different Data Client Library versions will likely fail. The Data Client LIbrary will throw an exception and, depending upon how your application handles this exception, could lead to an application crash or downstream processing failure. This adverse behavior is due to inconsistent compression and decompression of the data driven by the different Data Client Library versions. 2.10 introduces more tolerant behavior which correctly detects if stream data is compressed and handles it correctly.
Workaround: In the case where you are using compressed Stream layers and streaming messages smaller than 2MB, use the 2.8 SDK until you have confirmed that all of your customers are using at least the 2.10 SDK where this Data Client Library issue is resolved, then upgrade to the 2.10 version for the writing aspects of your workflow.
Issue: Searching for a schema in the Portal using the old HRN construct will return only the latest version of the schema. The Portal will currently not show older versions tied to the old HRN.
Workaround: Search for schemas using the new HRN construct OR lookup older versions of schemas by old HRN construct using the OLP CLI.
Issue: Visualization of Index layer data is not yet supported
Pipelines
Changed: In batch-2.1.0 run-time environment for batch pipelines, following libraries were upgraded in order to align with critical Data Client Library changes in the Platform SDK:
- com.squareup.okhttp3:okhttp from version 3.8.1 to version 3.12.10
- com.squareup.okio:okio from version 1.13.0 to version 1.15.0
Issue: A pipeline failure or exception can sometimes take several minutes to respond.
Issue: Pipelines can still be activated after a catalog is deleted.
Workaround: The pipeline will fail when it starts running and will show an error message about the missing catalog. Re-check the missing catalog or use a different catalog.
Issue: If several pipelines are consuming data from the same Stream layer and belong to the same Group (pipeline permissions are managed via a Group), then each of those pipelines will only receive a subset of the messages from the stream. This is because, by default, the pipelines share the same Application ID.
Workaround: Use the Data Client Library to configure your pipelines to consume from a single stream: If your pipelines/applications use the Direct Kafka connector, you can specify a Kafka Consumer Group ID per pipeline/application. If the Kafka consumer group IDs are unique, the pipelines/applications will be able to consume all the messages from the stream.
If your pipelines use the HTTP connector, we recommend you to create a new Group for each pipeline/application, each with its own Application ID.
Issue: The Pipeline Status Dashboard in Grafana can be edited by users. Any changes made by the user will be lost when updates are published in future releases because users will not be able to edit the dashboard in a future release.
Workaround: Duplicate the dashboard or create a new dashboard.
Issue: For Stream pipeline versions running with the high-availability mode, in a rare scenario, the selection of the primary Job Manager fails.
Workaround: Restart the stream pipeline.
HERE Content
Added: additions to existing layers in the HERE Map Content catalog:
- Added alt_category attribute in Place message (found in layers: HERE Essential Places, HERE Places and Building Footprints)
- Added LocationAddressType for the Address Locations layer
- Added MatchLevelAttribute in the Places layer.
- Added Vehicle Checkpoint condition to Advanced Navigation layer
- Added Walking path and Bicycle path to supplemental geometry enum (found in the Navigation Attributes layer)
- Added exception list for the political view in AdministrativeContext (found in layers: Address Attributes and Administrative Places)
- Added BicyclePathAttribute and deprecated DedicatedBicyclePathAttribute from Bicycle Attributes layer
Deprecated: Within the Real Time Traffic catalog (https://platform.here.com/data/hrn:here:data::olp-here:olp-traffic-1) the Incidents2.4 layer is being deprecated and has been labeled as such since August 2019. This is a step in the effort to migrate to the new Delta Incidents layer which is now available in the same catalog. There will no longer be new incidents published in the Incidents2.4 layer by April 30, 2020. Once this happens, volatile data will no longer have incidents after 1hr of their time-to-live. As this will be a schema change, there are instructions in the layer description on how to migrate.
Location Services
Issue: Lack of usage reporting for Location Services released in version 2.10 (Routing, Search, Transit, and Vector Tiles Service)
Workaround: Usage is being tracked at the service level. Following the 2.12 release wherein usage reporting is expected to be in place, customers may request usage summaries for usage incurred between the 2.10 release and 2.12.
Marketplace (Not available in China)
Issue: Users do not receive stream data usage metrics when reading or writing data from Kafka Direct.
Workaround: When writing data into a Stream layer, you must use the Ingest API to receive usage metrics. When reading data, you must use the Data Client Library, configured to use the HTTP connector type, to receive usage metrics and read data from a Stream layer.
Issue: When the Technical Accounting component is busy, the server can lose usage metrics.
Workaround: If you suspect you are losing usage metrics, contact HERE technical support for assistance rerunning queries and validating data.
Issue: Projects and all resources in a Project are designed for use only in Workspace and are unavailable for use in Marketplace. For example, a catalog created in a Platform Project can only be used in that Project. It cannot be marked as "Marketplace ready" and cannot be listed in the Marketplace.
Workaround: Do not create catalogs in a Project when they are intended for use in the Marketplace.
Summary of active deprecation notices across all components
No. |
Feature Summary |
Deprecation Period Announced (Platform Release) |
Deprecation Period Announced (Month) |
Deprecation Period End |
||
1 |
OrgID added to Catalog HRN (RoW) |
2.9 (ROW) 2.10 (China) |
November 2019 |
July 31, 2020 |
||
|
Deprecation Summary: |
Catalog HRNs without OrgID will no longer be supported in any way after July 31, 2020.
|
||||
2 |
OrgID added to Schema HRN (Global) |
2.10 |
December 2019 |
July 31, 2020 |
||
|
Deprecation Summary: |
References to pre-existing schemas created before this feature release will work with either new or old HRNs so those references are not impacted. New schemas created as of this feature release will support both HRN constructs for a period of (6) months or until June 30, 2020. After this time, only the new HRN construct will be supported. |
||||
3 |
Data Client Library data compression behavior |
2.10 |
December 2019 |
June 20, 2020 |
||
|
Deprecation Summary: |
As of June 30, 2020, all data streaming related workflows must be updated to use at least SDK version 2.9. The Data Client Library behavior as it relates to compressing and decompressing streaming data will strictly adhere to stream layer configuration settings. |
||||
4 |
Spark-ds-connector replaced by SDK for Java and Scala Spark Connector |
2.12 |
February 2020 |
August 19, 2020 |
||
|
Deprecation Summary: |
The spark-ds-connector will be deprecated (6) months from this release on August 19, 2020. Please upgrade to the latest SDK for Python version before then to get the latest SDK for Java and Scala Spark Connector. |
||||
5 |
Batch-2.0.0 run-time environment for Pipelines |
2.12 |
February 2020 |
August 19, 2020 |
||
|
Deprecation Summary: |
Batch-2.0.0 run-time environment for Batch pipelines is now deprecated. Existing Batch pipelines that use the Batch-2.0.0 run-time environment will continue to operate normally until August 19, 2020. During this period, Batch-2.0.0 run-time environment will receive security patches only. For this period, to continue developing pipelines with the Batch-2.0.0 environment, please use OLP SDK 2.11 or older. After August 19, 2020 we will remove the Batch-2.0.0 run-time environment and the pipelines still using it will be canceled. We recommend that you migrate your Batch Pipelines to the Batch-2.1.0 run-time environment to utilize the latest functionality and improvements. |
||||
6 |
Schema validation to be added |
2.13 |
March 2020 |
September 30, 2020 |
||
|
Deprecation Summary: |
For security reasons, the platform will start validating schema reference changes in layer configurations as of September 30, 2020. Schema validation will check if the user or application trying to make a layer configuration change indeed has at least read access to the existing schema associated with that layer (i.e. a user or application cannot reference or use a schema they do not have access to). If a non-existing or non-accessible schema is associated with any layer after this date, any attempt to update any configurations of that layer will fail. Please ensure all layers refer only to real, existing schemas, or contain no schema reference at all before September 30, 2020. It is possible to use the Config API to remove or altogether change schemas associated with layers to resolve these invalid schema/layer associations. Also, any CI/CD jobs referencing non-existing or non-accessible schemas will need to be updated by this date or they will fail. |
||||
7 |
Customizable Volatile layer storage capacity and redundancy configurations |
2.14 |
April 2020 |
October 30, 2020 |
||
|
Deprecation Summary: |
The Volatile layer configuration option to set storage capacity as a "Package Type" will be deprecated within (6) months or by October 30, 2020. All customers should deprecate their existing volatile layers and create new volatile layers with these new configurations within (6) months of this feature release or by October 30, 2020. |
||||
Have your say
Sign up for our newsletter
Why sign up:
- Latest offers and discounts
- Tailored content delivered weekly
- Exclusive events
- One click to unsubscribe