Log Files

If enabled, K3s single-instance IVAAP installations will write out all logs to files to

/opt/ivaap/ivaap-volumes/logs

in separate directories by component:

$ ls -1 /opt/ivaap/ivaap-volumes/logs/
activemq
backend
adminserver
postgres
proxy
scheduledtasks

Inside each directory there will be a live .log file, as well as rotated out files from previous history. INT recommends shipping these logs to your monitoring tool of choice for production deployments, either by scraping these files with something like Logstash/FileBeat or similar, or capturing them from the container outputs. When contacting SLB Support, please include the relevant log files for all components.

Capturing Versions

It’s often useful to check the versions of individual components, either for requesting support, comparing deltas before updating, or verifying against other documentation. Here are some various methods to find relevant versions/tags.

Helm Chart Version

Included in all IVAAP Helm Templates will be Chart.yaml. This file will contain not only the IVAAP version, but also the version of the Helm Charts provided.

user@linux:/opt/ivaap/IVAAPHelmTemplate$ cat Chart.yaml 
apiVersion: v2
name: ivaap
description: Helm chart for deploying IVAAP
type: application
version: 1.1.0
appVersion: "2025.1.0"
maintainers:
  - name: Support
    email: customersupport.digital@slb.com
home: https://slb.com/
sources:
  - https://gerrit.int.com/IVAAPHelmTemplate

This will give SLB most of the information needed. However, sometimes updated builds may be provided piecemeal without being provided a full new helm template. If there are image tags in the your deployment yaml that are overriding the default tags in the helm template, this information will be important to pass along to SLB support.

Image Tags

Image tags can be found in the Helm Template file values.yaml, and possibly within your configured deployment yaml file. For example you will see information for each node deployed as either a corenode or as a additional enabled datanode with the tag and reponame.

Example tags:

ivaapBackendNodes
  coreNodes:
    seednode:
      repoName: ivaap/backend/seednode
      tag: seednode-3.0-3-0af410b-20250620T150944Z
  dataNodes:
    dataimportnode:
      enabled: false
      repoName: ivaap/backend/dataimportnode
      tag: dataimportnode-3.0-3-0af410b-20250620T150944Z

A convinient script is provided in the helm template to collect a list of the images within the template values.yaml file:

user@linux:/opt/ivaap/IVAAPHelmTemplate$ ./collect-image-list.sh
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/activemq:activemq-6.1.7-96e1cba9-202506251620Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/adminserver:adminserver-3.0.42-569ce02-eclipse-temurin-21-jdk-alpine-202506262109Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/adminnode:adminnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/blobstoragenode:blobstoragenode-3.0-3-bb04d54-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/cloudstoragenode:cloudstoragenode-3.0-3-fd24b41-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/dataimportnode:dataimportnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/epsgnode:epsgnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesmasternode:geofilesmasternode-3.0-2-f38d7a2-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesreservoirsnode:geofilesreservoirsnode-3.0-2-f38d7a2-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesseismicnode:geofilesseismicnode-3.0-2-f38d7a2-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/messagingnode:messagingnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/mongonode:mongonode-3.0-6-77b1f74-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/mqgatewaynode:mqgatewaynode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/opensdublobstorager3node:opensdublobstorager3node-3.0-4-6d06752-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/opensducloudstoragenode:opensducloudstoragenode-3.0-4-6d06752-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/opensdus3r3node:opensdus3r3node-3.0-4-6d06752-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/playnode:playnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/ppdmnode:ppdmnode-3.0-4-5fcc873-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/s3node:s3node-3.0-4-7b59464-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/seednode:seednode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/witsmlnode:witsmlnode-3.0-1-c2f557d-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/admin-client:ivaap_admin-3.0.1
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:ivaap-dashboard-publish-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:ivaap-dashboard-publish-external-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:osdu-publish-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:ivaap-dashboard-standard-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:ivaap-dashboard-standard-external-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:osdu-standard-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/infinispan:infinispan-15.1.7.Final-fa466eec-20250623T152424Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/proxy:proxy-2024.0-cce98866-20250627T135324Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/task-release:witsmlliveupdatetask_release-3.0.1-1511246-202507091432-20250709T143406Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/task-release:witsmlpollingtask_release-3.0.1-1511246-202507091432-20250709T143406Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/task-release:witsmlwelltrajectorypollingtask_release-3.0.1-1511246-202507091432-20250709T143406Z

However, this will not tell the whole picture. A more accurate description of your deployment would be to use the icvt alias included with ivaap-helpers. icvt <namespace> command to help find all current versions that are deployed.

user@linux:/opt/ivaap/IVAAPHelmTemplate$ icvt ivaap
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/adminserver:adminserver-3.0.42-569ce02-eclipse-temurin-21-jdk-alpine-202506262058Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/activemq:activemq-6.1.7-96e1cba9-202506251620Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/admin-client:ivaap_admin-3.0.1
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/adminnode:adminnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/epsgnode:epsgnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/messagingnode:messagingnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/mqgatewaynode:mqgatewaynode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/playnode:playnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/seednode:seednode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesmasternode:geofilesmasternode-3.0-2-f38d7a2-20250626T003400Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesreservoirsnode:geofilesreservoirsnode-3.0-2-f38d7a2-20250626T003400Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesseismicnode:geofilesseismicnode-3.0-2-f38d7a2-20250626T003400Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:ivaap-dashboard-standard-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:ivaap-dashboard-publish-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/proxy:proxy-2024.0-cce98866-20250627T135324Z

Specific JAR Commit Hashes

The commit hash is also embedded in each individual IVAAP jars. This can be useful to trace specific issues for SLB developers.

To find the jar filename in a container within a backend deployment pod follow these steps to exec into it and grep for the right jar. Example:

[user@ivaap IVAAPHelmTemplate]$ kubectl get pods -n ivaap
NAME                                                  READY   STATUS    RESTARTS   AGE
adminserver-deployment-759fb78996-dmdld               1/1     Running   0          21d
ivaap-activemq-deployment-6b6956684f-9v5c7            1/1     Running   0          21d
ivaap-admin-deployment-57669dd46d-khvq7               1/1     Running   0          21d
ivaap-backend-deployment-6f5575b6ff-9wwnh             9/9     Running   0          21d
ivaap-dashboard-deployment-96b65978f-j6mgl            1/1     Running   0          21d
ivaap-dashboard-publish-deployment-6bc4d97ddf-pqjdb   1/1     Running   0          21d
ivaap-proxy-deployment-6875dcf849-9nbtq               1/1     Running   0          21d

[user@ivaap IVAAPHelmTemplate]$ kubectl get pod ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -o jsonpath="{.spec.containers[*].name}"
adminnode epsgnode messagingnode mqgatewaynode playnode seednode geofilesmasternode geofilesreservoirsnode geofilesseismicnode


[user@ivaap IVAAPHelmTemplate]$ kubectl exec -it ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -c messagingnode -- /bin/bash -c "ls /opt/ivaap/ivaap-playserver/deployment/ivaapnode/lib" | grep -i coredata
com.interactive.ivaap.IVAAPCoreData-3.0.1.jar

Then with the appropriate filename, you can get the commit hash of the jar:

$ unzip -q -c /opt/ivaap/ivaap-playserver/deployment/ivaapnode/lib/com.interactive.ivaap.IVAAPCoreData-3.0.1.jar META-INF/MANIFEST.MF | grep -i revision | cut -d '' -f 2
Git-Revision: 1511246

This also works for the scheduled tasks:

$ unzip -q -c /opt/ivaap/ivaap-playserver/scheduledtasks/ivaaptask/lib/IVAAPWitsmlAdapter-3.0.1.jar META-INF/MANIFEST.MF | grep -i revision | cut -d '' -f 2
Git-Revision: 4f3c49c

HTTP APIs

Finally, HTTP APIs also show version information for many components

For the Adminserver, IVAAPServer v2, the endpoint is: /IVAAPServer/api/v2/about

$ curl -s https://<DNS>/IVAAPServer/api/v2/about | jq .
{
  "name": "IVAAP Data Backend",
  "version": "3.0.42-569ce02",
  "majorVersion": "3",
  "minorVersion": "0",
  "buildVersion": "42-569ce02",
  "database": {
    "version": "PostgreSQL 15.10",
    "description": "PostgreSQL 15.10 (Ubuntu 15.10-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit"
  },
  "localTime": {
    "day": 5,
    "dayOfYear": 217,
    "month": 8,
    "year": 2025,
    "hour": 19,
    "minute": 25,
    "second": 15,
    "millisecond": 950
  },
  "localTimeZone": {
    "id": "GMT",
    "name": "(GMT0:00) GMT"
  },
  "api": {
    "versions": [
      {
        "id": "v2",
        "links": [
          {
            "rel": "OpenApi Specification",
            "name": "OpenApi Specifications",
            "relEntity": "v1/schema/int/openapispec",
            "children": false,
            "hasProjectEntityChildren": false,
            "isProjectEntity": false,
            "href": "/IVAAPServer/api/v2/openapispecs"
          },
...

For the backend: /ivaap/api/v1/cluster/services/versions

$ curl -s https://<DNS>/ivaap/api/v1/cluster/services/versions | jq .
{
  "timestamp": 1754422214432,
  "configuredServiceRoleCount": 7,
  "configuredServiceRoleNames": "admin,epsg,geofiles,geofiles::reservoirs,geofiles::seismic,messaging,mqgateway",
  "servicesVersionInfoCount": 7,
  "servicesVersionInfoValues": [
    {
      "name": "admin",
      "UUID": "990816d4-88f1-4a5a-9cdc-435647876ed2",
      "description": "Non-datasource specific node used for global cluster requests.",
      "versionMajor": "3",
      "versionMinor": "0",
      "versionBuild": "3-0af410b"
    },
    {
      "name": "epsg",
      "UUID": "819c378a-e654-491c-ab5f-1ecdc4ce1137",
      "versionMajor": "3",
      "versionMinor": "0",
      "versionBuild": "3-0af410b"
    },
    {
      "name": "geofiles",
      "UUID": "15b1bf3d-9eaa-4333-81c0-a80d1d4a1a3a",
      "description": "This node is packaged with native code: \\\\n IVAAPNativeLibs: f3e655e9 \\\\n IVAAPOpenSeismicNativeLibs: ",
      "versionMajor": "3",
      "versionMinor": "0",
      "versionBuild": "2-f38d7a2"
    },
...

Get requests to the above (or just entering them into the browser's URL bar) will return versions of running components

Backend Health API

Generally speaking, IVAAP deployment will be healthy if all pods are running and none of them are in a restart loop. For the backend nodes, ivaap-helpers includes an alias for checking the backend health API. The syntax is ibhealth <namespace>:

user@linux:/opt/ivaap/IVAAPHelmTemplate$ ibhealth ivaap
{
  "timestamp" : 1755618351429,
  "health" : "healthy",
  "available" : "admin,blobstorage,cloudstorage,dataimport,epsg,geofiles,geofiles::reservoirs,geofiles::seismic,messaging,mongo,mqgateway,ppdm,s3,witsml",
  "activeMQConnectionStatus" : {
    "healthy" : true
  }
}

Component-Specific Health Logic

• Postgres
  • Check the testresults API for adminserver connection to the postgres database
  • /IVAAPServer/api/v2/testresults

    {
      "from": {
        "actor": "akka://DefaultActorSystemControllerActorSystem:TestDataSourceByNameJsonActor:TestDataSourceByNameResponse for TestDataSourceByNameRequest{sourceType=team, sourceName=main}"
      },
      "success": true
    }
    

• Adminserver
  • Port: 8080
  • Internal Endpoint
    • <DNS>/IVAAPServer/api/v2/testresults
    • <DNS>/IVAAPServer/api/v2/servicetypes/trusts/services/main/testresults
    • <DNS>/IVAAPServer/api/v2/servicetypes/about/services/main/testresults
    • <DNS>/IVAAPServer/api/v2/servicetypes/datasourcescontrol/services/main/testresults
• Reverse Proxy (NGINX)
  • Ports: 80 (HTTP), 443 (HTTPS)
  • Internal Endpoint
    • ivaap-backend-service:80
    • ivaap-adminserver-service:80
    • ivaap-dashboard-service:80
    • ivaap-dashboard-publish-service:80
    • ivaap-admin-service:80
• ActiveMQ
  • Ports: 61616 (TCP), 8161 (HTTP)
• Seednode
  • Port: 2550 (Default, envar based)
  • Internal Endpoint
    • nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT
  • Health Check: Seednode checks connectivity to its own cluster network port, making sure it is listening and ready for new connections (other nodes) to connect to it.
• Playnode
  • Port: 9000
  • Dependencies: Depends on the healthy status of Seednode and ActiveMQ. For smooth operation, ActiveMQ should be initialized before the backend node cluster to avoid WebSocket connection issues.
  • Internal Endpoint
    • nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT
    • ${IVAAP_COMMON_BACKEND_SERVER_HOST}api/v1/cluster/services/health
• MQGatewaynode
  • Dependencies: Like the playnode, MQGatewaynode also depends on the healthy status of Seednode and ActiveMQ. For smooth operation, ActiveMQ should be initialized before the backend node cluster to avoid WebSocket connection issues.
  • Internal Endpoint
    • nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT
• Other backend nodes (Adminnode, Epsgnode, Messagingnode, and all Data Connector Nodes)
  • Dependencies: All depend on the healthy status of Seednode.
  • Internal Endpoint
    • nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT

Manual Health APIs and Metrics

To verify the state of IVAAP, there are a few checks to be made to its components. These can be made in a browser, or the server’s shell. If none of these are working, even from the local shell, restart and check the proxy (NGINX) service’s logs, as all of these calls depend on it being operational. Some of these APIs are used in the container level health checks described in the Container Health Checks Chapter

#Restart the reverse proxy and check logs
kubectl delete pod ivaap-proxy-deployment-6875dcf849-9nbtq -n ivaap
kubectl logs ivaap-proxy-deployment-6875dcf849-9nbtq -n ivaap

Admin Server

The first Adminserver health check API can be found at /IVAAPServer/api/v2/about

/IVAAPServer/api/v2/servicetypes/datasourcescontrol/services/main/testresults

Note that unless these envars are set true, these APIs will require a valid bearer token:

IVAAP_DISABLE_SETUP_TEST_CREDENTIALS_CONTROL=true
IVAAP_DISABLE_SETUP_TEST_ENTITLEMENTS_CONTROL=true

The above environment variables are not safe for production. Please ensure to use these variables with caution and for troubleshooting purposes only!

Example (note ‘success:true’ at bottom)

$ curl -s https://<DNS>/IVAAPServer/api/v2/servicetypes/datasourcescontrol/services/main/testresults | jq 
{
  "from": {
    "actor": "pekko://DefaultActorSystemControllerActorSystem:TestHostedServiceByNameJsonActor:TestHostedServiceByNameResponse for TestHostedServiceByNameRequest{serviceType=datasourcescontrol, serviceName=main, sourceType=team}"
  },
  "success": true
}

Backend Admin Node

The first backend health check verifies cluster activity: https://<DNS>/ivaap/api/about

[user@ivaap IVAAPHelmTemplate]$ curl -s https://<DNS>/ivaap/api/about | jq
{
  "name": "IVAAP Data Backend",
  "version": "3.0.3-0af410b",
  "majorVersion": "3",
  "minorVersion": "0",
  "buildVersion": "3-0af410b",
  "localTime": {
    "day": 5,
    "dayOfYear": 217,
    "month": 8,
    "year": 2025,
    "hour": 20,
    "minute": 43,
    "second": 49,
    "millisecond": 366
  },
  "localTimeZone": {
    "id": "GMT",
    "name": "(GMT0:00) GMT"
  },
  "licenses": {
    "IVAAPServer": {
      "type": "expires",
      "state": "valid",
      "expiration": {
        "daysUntilExpiration": 20,
        "day": 25,
        "dayOfYear": 237,
        "month": 8,
        "year": 2025
      }
    }
  },
  "api": {
    "versions": [

This check shows the version of the admin node deployed. If this response comes back as "no available license found," check the license envar or contact SLB

Backend Data Nodes

After verifying the backend data node, you can check other parameters about the backend nodes with the following. This will return a health status, with available nodes as well as any nodes that are available but not configured.

$ curl -s https://<DNS>/ivaap/api/v1/cluster/services/health | jq
{
  "timestamp": 1650405721028,
  "health": "healthy",
  "available": "admin,dataimport,eia,epsg,geofiles,messaging,mongo,mqgateway,s3,witsml",
  "activeMQConnectionStatus": {
    "healthy": true
  }
}

As you can see in the example above, this API reports which nodes have joined and are active in the cluster, as well as the status of the ActiveMQ connection. A healthy ActiveMQ connection status is paramount in real-time updates for both projects and data.

Another useful request is https://<DNS>/ivaap/api/v1/cluster/services/metrics. This response includes detailed real-time performance metrics for each individual node:

[user@ivaap IVAAPHelmTemplate]$ curl -s https://<DNS>/ivaap/api/v1/cluster/services/metrics | jq
{
  "timestamp": 1754427603633,
  "configuredServiceRoleCount": 7,
  "configuredServiceRoleNames": "admin,epsg,geofiles,geofiles::reservoirs,geofiles::seismic,messaging,mqgateway",
  "servicesMetricsCount": 8,
  "servicesMetricsValues": [
    {
      "name": "admin",
      "UUID": "990816d4-88f1-4a5a-9cdc-435647876ed2",
      "startTime": "Mon Jul 14 20:32:15 GMT 2025",
      "startTimestamp": 1752525135057,
      "upTime": "22 days, 0 hours and 27 minutes",
When reaching out to INT support, capture versions and logs to help track down the issues.      "processCpuLoadPct": 0.13464991023339318,
      "processCpuTime": 17080230000000,
      "usedHeapKB": 318469,
      "committedHeapKB": 524288,
      "threadCount": 55,
      "daemonThreadCount": 12,
      "peakThreadCount": 62,
      "timestamp": 1754427603618
    },
...

Finally, the last check requires a valid authorization token to be included, so it is best to use the IVAAP Admin Client and verify the data sources are available at https://<DNS>/admin/#/ClusterActivities. This page serves as a front-end visualization of some of the preceding health checks.

Resetting passwords or reactivating locked accounts

Sometimes it may be necessary to directly manipulate the postgres database if all super admin accounts are locked. This can happen if too many incorrect login attempts are made. The adminserver container has baked-in functions in it's bashrc to assist with database manipulation. For example, running the command listSuperAdmins inside the adminserver pod will list all super admin users, including their active status:

user@linux:~$ kubectl exec -it adminserver-deployment-6b4bfb4497-c4bp6 -n ivaap  -- /bin/bash
adminserver-deployment-6b4bfb4497-c4bp6:/opt/ivaap/adminserver$ listSuperAdmins
                 guid                 | fname | lname |       email       | deleted | active |     username      |
--------------------------------------+-------+-------+-------------------+---------+--------+-------------------+
 415b1d1c-78ba-41b2-86a3-37067326156e | IVAAP | Root  | ivaaproot@int.com | f       | t      | ivaaproot@int.com |
(1 row)

If this active status is f for false, the command resetAllSuperAdminActive can be used to set the status back to true. This should allow the user to login as super admin again. Note that this does reset all super admin's status back to true.

These scripts are also availabe in the ivaap-helpers repo under scripts/ivaap-postgresdb-utils directory

Enable Additional Logging Levels (Debug)

At times, the default log levels are not sufficient to help diagnose issues. The following steps can be used to enable additional DEBUG logging in the different IVAAP components.

IVAAPData Backend Nodes and Tasks

The backend data nodes and tasks can adjust logging levels via the logback.xml configuration files in each node directory

[user@ivaap IVAAPHelmTemplate]$ kubectl exec -it ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -c adminnode -- /bin/bash
ivaap-backend-deployment-6f5575b6ff-9wwnh:/opt/ivaap/ivaap-playserver/deployment$ cd ivaapnode/conf/
ivaap-backend-deployment-6f5575b6ff-9wwnh:/opt/ivaap/ivaap-playserver/deployment/ivaapnode/conf$ vi logback.xml

For the tasks, they can be reached inside a container by:

[user@ivaap IVAAPHelmTemplate]$ kubectl exec -it ivaap-scheduledtask-deployment-5dc6494ff-fnlc5 -n ivaap -c witsmlpollingtask -- /bin/bash
/opt/ivaap/ivaap-playserver/scheduledtasks/ivaaptask $ cd conf
/opt/ivaap/ivaap-playserver/scheduledtasks/ivaaptask/conf $ vi logback.xml

In the xml file, the WARN values can be replaced with DEBUG, or lines for specific classes/wildcards to DEBUG and manipulate logs can be added very easily. No restart is required once changes are made to the file and saved; they will take effect immediately.

Capturing memory dumps (debugging)

Capturing java heap dumps can sometimes be useful to examine internal data allocations etc. INT may request you perform the steps below to output memory dumps to the relevant components’ logs directories. We recommend gzipping the resulting .hprof files before sending them to INT for analysis.

Before beginning, identify the container name for the target component you are wishing to create a jvm heap dump for. In the example below, the kubectl get pods -n ivaap command prints only the pod names within the ivaap namespace.cont Memory dumps can be captured from the “ivaap-backend-deployment”, “ivaap-scheduledtask-deployment”, and “adminserver-deployment” pods.

[user@ivaap IVAAPHelmTemplate]$ kubectl exec ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -c messagingnode -- bash -c 'pgrep java | xargs jmap -dump:format=b,file=/opt/ivaap/ivaap-playserver/deployment/logs/heap.hprof'
Dumping heap to /opt/ivaap/ivaap-playserver/deployment/logs/heap.hprof ...

This can then be copied out of the pod container onto your hosts file system.

kubectl cp ivaap/ivaap-backend-deployment-6f5575b6ff-9wwnh:/opt/ivaap/ivaap-playserver/deployment/logs/heap.hprof ./heap.hprof -c messagingnode

Common Issues with IVAAP Components

This section covers common pitfalls and bad states that can sometimes arise in various containerized environments.

502s in Browser

Sometimes, especially with older versions of our ingress proxy configuration, IVAAP components can become unavailable through the ingress proxy, despite being healthy internally. This can happen after a restart or update of individual components. In any case, often just restarting the proxy will resolve these issues:

kubectl delete pod ivaap-proxy-deployment-6875dcf849-9nbtq -n ivaap