Powered by Zoomin Software. For more details please contactZoomin

Data Client Library - Developer Guide

Filters

Clear All Filters
Product category
Technology
Doc type
Version
Product lifecycle
Clear All Filters
This publication

Data Client Library - Developer Guide: Create Catalogs

Share
Share to emailCopy article URL
Feedback
Print
Table of Contents
HomeData Client Library - Developer GuideCreate CatalogsCurrent page
Table of Contents

Create Catalogs

Share
Share to emailCopy article URL
Feedback
Print

To manage catalogs, add the data-client module as a dependency to your project.

Configure Catalogs

Catalog configurations include the properties listed in the following table. Mandatory properties are in bold.

Note

Catalog IDs
Catalog IDs are publicly visible since they are part of the catalog HRN. Do not include private or company confidential information when you specify a catalog ID. Catalog names are private by default, which means you can add your private or confidential information in this field.

Layer configurations include the properties listed in the table below.

Versioned layer configuration

Versioned layers have the following additional configuration fields.

Property Description
secureHandles A boolean flag which indicates if the layer metadata and blob are in mode of secure data handles support. It requires catalog area scheme to be set and supports only heretile partitioning of the layer.
subsetRestrictions A boolean flag which indicates if the subsetRestrictions for a layer is enabled or not. subsetRestrictions can only be enabled for layers of catalogs that are defined within projects. It requires area scheme of the catalog being set. Supported layer subsetRestrictions are enabled by default when subsetRestrictions is enabled for the catalog. Layer subsetRestrictions requires catalog subsetRestrictions to be enabled. If catalog subsetRestrictions are disabled, supported layer's subsetRestrictions will be disabled by default. Supported only for layers with heretile partitioning.

Volatile layer configuration

Volatile layers have the following additional configuration fields.

Index layer configuration

Index layers have the following additional configuration fields.

Property Description
indexDefinitions The collection of index key definitions to index published partitions.
ttl Time to Live of indexed data in index layer. If not specified it defaults to 7 days.

Each index key definition contains the following configuration fields:

Supported index key types:

The TimeWindow index key is required. Index layers can have at most a single TimeWindow and a single HereTile index key. This means that Index layers can have exactly one TimeWindow index key and only one optional HereTile index key.

Below are the possible TTL values:

Stream layer configuration

Stream layers have the following additional configuration fields.

Note

General recommendation for stream layer properties
(dataInThroughputKbps and dataOutThroughputKbps and parallelization) is to provide all of them in the request.

To preserve backward compatibility, there are few corner cases:

  1. User should provide either dataInThroughputKbps and dataOutThroughputKbps or parallelization.
  2. If all 3 fields are provided, then we use them as is.
  3. If all stream properties are missing, then default values dataInThroughputKbps = 1000, dataOutThroughputKbps = 4000, parallelization = 4 will be set

Interactive Map layer configuration

Interactive Map layers have the following additional configuration fields.

Property Description
Searchable Properties Interactive map layers can be configured with 'Searchable Properties'. This is a list of strings containing the properties you want to index for faster searching. Searchable properties can be updated at any time. A maximum of eight searchable properties can be added.

Automatic Version Deletion Configuration

Warning

Data loss
Deleting Catalog versions will delete partition metadata AND data associated with those versions, impacting all versioned layers in the catalog. The partition metadata and data will be irrecoverable. Any partition metadata and data from deleted versions that is still used in current, non-deleted versions will not be deleted so that non-deleted versions remain functional.

Supported automaticVersionDeletion types:

Property Description
numberOfVersionsToKeep If set and once the max number of versions is reached, every time a new version is created, the oldest version will be deleted.

Note

For instructions on how to disable automatic version deletion, see Disable an Automatic Version Deletion

Create Catalogs

To configure and create catalogs, add the following:


    val catalogId = "creating-catalog-test-" + someIdentifierSuffix
    val catalogConfig =
      WritableCatalogConfiguration(
        id = catalogId,
        name = catalogId,
        summary = "This is a catalog summary.",
        description = "This is what the catalog is for.",
        tags = Set("tag1", "tag2", "tag3"),
        automaticVersionDeletion =
          Some(AutomaticVersionDeletion.builder.withNumberOfVersionsToKeep(10L).build),
        areaScheme = Some(HRN.fromString("hrn:here:areascheme::olp-here:default-countries")),
        subsetRestrictions = Some(true),
        layers = Seq(
          WritableLayer(
            id = "layer-0",
            name = "INFO",
            summary = "This is a layer summary.",
            description = "This is a layer description.",
            layerType = VersionedLayerType(),
            partitioning = GenericPartitioning,
            volume = Volumes.Durable,
            contentType = "application/x-protobuf",
            digest = Some(DigestAlgorithm.MD5),
            crc = Some(CrcAlgorithm.CRC32C)
          ),
          WritableLayer(
            id = "layer-1",
            name = "SCHEMAS",
            summary = "This is a layer summary.",
            description = "This is a layer description.",
            layerType = VersionedLayerType(),
            partitioning = GenericPartitioning,
            volume = Volumes.Durable,
            contentType = "application/x-protobuf"
          ),
          WritableLayer(
            id = "layer-2",
            name = "VOLATILEDATA",
            summary = "This is a layer summary.",
            description = "This is a layer description.",
            layerType =
              VolatileLayerType(secureHandles = Some(true), subsetRestrictions = Some(true)),
            partitioning = HereTilePartitioning(tileLevels = 1 :: Nil),
            volume = Volumes.Durable,
            contentType = "application/octet-stream",
            contentEncoding = Some(ContentEncoding.gzip),
            coverage = Some(Coverage(Set[String]("DE", "US")))
          ),
          WritableLayer(
            id = "layer-3",
            name = "INDEX",
            summary = "This is a layer summary.",
            description = "This is a layer description.",
            layerType = IndexLayerType(
              indexDefinitions = Seq(
                IndexDefinition("someIntKey", IndexType.Int),
                IndexDefinition("someStringKey", IndexType.String),
                IndexDefinition("someTimeWindowKey",
                                IndexType.TimeWindow,
                                duration = Some(3600000)),
                IndexDefinition("someHereTileKey", IndexType.HereTile, zoomLevel = Some(8))
              ),
              ttl = Ttl.OneMonth
            ),
            partitioning = NoPartitioning,
            volume = Volumes.Durable,
            contentType = "application/octet-stream",
            digest = Some(DigestAlgorithm.SHA256),
            crc = Some(CrcAlgorithm.CRC32C)
          ),
          WritableLayer(
            id = "layer-4",
            name = "OBJECTSTORE",
            summary = "This is layer summary",
            description = "This is layer description",
            layerType = ObjectStoreLayerType(),
            partitioning = NoPartitioning,
            volume = Volumes.Durable
          ),
          WritableLayer(
            id = "layer-5",
            name = "INTERACTIVEMAP",
            summary = "This is layer summary",
            description = "This is layer description",
            layerType = InteractiveMapLayerType(
              interactiveMapProperties = Some(
                InteractiveMapProperties(searchableProperties = Seq("abc"),
                                         versionsToKeep = Some(123L)))
            ),
            partitioning = NoPartitioning,
            volume = Volumes.Durable
          )
        )
      )

    val adminApi = DataClient().adminApi()
    
      adminApi.createCatalog(catalogConfig).flatMap { hrn =>
        log.info(s"Created new catalog `$hrn`")
        processCreatedCatalog(hrn)
      }
      

    String catalogId = "creating-catalog-test-" + someIdentifierSuffix;
    WritableCatalogConfiguration catalogConfig =
        new WritableCatalogConfiguration.Builder()
            .withId(catalogId)
            .withName(catalogId)
            .withSummary("This is a catalog summary.")
            .withDescription("This is what the catalog is for.")
            .withTags(new HashSet<>(Arrays.asList("tag1", "tag2", "tag3")))
            .withAutomaticVersionDeletion(
                AutomaticVersionDeletion.builder().withNumberOfVersionsToKeep(10L).build())
            .withLayers(
                Arrays.asList(
                    new WritableLayer.Builder()
                        .withId("layer-0")
                        .withName("INFO")
                        .withSummary("This is a layer summary.")
                        .withDescription("This is a layer description.")
                        .withLayerType(VersionedLayerType.defaultInstance())
                        .withPartitioning(Partitioning.Generic())
                        .withVolume(Volumes.Durable())
                        .withContentType("application/x-protobuf")
                        .withDigest(DigestAlgorithm.SHA256())
                        .withCrc(CrcAlgorithm.CRC32C()),
                    new WritableLayer.Builder()
                        .withId("layer-1")
                        .withName("SCHEMAS")
                        .withSummary("This is a layer summary.")
                        .withDescription("This is a layer description.")
                        .withLayerType(VersionedLayerType.defaultInstance())
                        .withPartitioning(Partitioning.Generic())
                        .withVolume(Volumes.Durable())
                        .withContentType("application/x-protobuf"),
                    new WritableLayer.Builder()
                        .withId("layer-2")
                        .withName("INDEX")
                        .withSummary("This is a layer summary.")
                        .withDescription("This is a layer description.")
                        .withLayerType(
                            new IndexLayerType.Builder()
                                .addIndexDefinition(
                                    new IndexDefinition.Builder()
                                        .withName("someIntKey")
                                        .withIndexType(IndexType.Int)
                                        .build())
                                .addIndexDefinition(
                                    new IndexDefinition.Builder()
                                        .withName("someStringKey")
                                        .withIndexType(IndexType.String)
                                        .build())
                                .addIndexDefinition(
                                    new IndexDefinition.Builder()
                                        .withName("someTimeWindowKey")
                                        .withIndexType(IndexType.TimeWindow)
                                        .withDuration(3600000L)
                                        .build())
                                .addIndexDefinition(
                                    new IndexDefinition.Builder()
                                        .withName("someHereTileKey")
                                        .withIndexType(IndexType.HereTile)
                                        .withZoomLevel(8)
                                        .build())
                                .withTtl(Ttl.OneMonth)
                                .build())
                        .withPartitioning(Partitioning.NoPartition())
                        .withVolume(Volumes.Durable())
                        .withContentType("application/x-protobuf"),
                    new WritableLayer.Builder()
                        .withId("layer-3")
                        .withName("OBJECTSTORE")
                        .withSummary("This is layer summary")
                        .withDescription("This is layer description")
                        .withLayerType(ObjectStoreLayerType.defaultInstance())
                        .withPartitioning(Partitioning.NoPartition())
                        .withVolume(Volumes.Durable()),
                    new WritableLayer.Builder()
                        .withId("layer-4")
                        .withName("INTERACTIVEMAP")
                        .withSummary("This is layer summary")
                        .withDescription("This is layer description")
                        .withLayerType(
                            new InteractiveMapLayerType.Builder()
                                .withInteractiveMapProperties(
                                    new InteractiveMapProperties.Builder()
                                        .withSearchableProperties(
                                            Arrays.asList("some-property1", "some-property-2"))
                                        .build())
                                .build())
                        .withPartitioning(Partitioning.NoPartition())
                        .withVolume(Volumes.Durable())
                        .withContentType("application/geo+json")))
            .build();

    AdminApi adminApi = DataClient.get(myActorSystem).adminApi();
    
          adminApi
              .createCatalog(catalogConfig)
              .thenApply(
                  hrn -> {
                    log.info("Created new catalog `" + hrn + "`");
                    return processCreatedCatalog(hrn);
                  });

Note

It may take several minutes for a new catalog to appear in the list of catalogs returned by the Data Client Library, CLI, REST API, and the list displayed on the HERE platform portal.

In This Article
Was this article helpful?
YesNo
TitleResults for “How to create a CRG?”Also Available inAlert