Analytics Sync Process

This article provides a detailed description of all the features and options associated with the AI Document Sync process

What does AI Document Sync do?

The AI Document Sync process consists of many different analytics processing tasks. Many of these are similar to those previously done in Brainspace and Reveal AI. From a high level:

• Sync begins with text and metadata ingestion into the AI processing queue.
• Next, the build process runs the analytics (see below), and
• Outputs reports that are indexed to be used by visualizations, searching, and more.

The process also extracts AI vectors and metadata features that are processed for use in Supervised Learning classifiers.

Some examples of the AI analytics tasks that are run include:

• Threading and unique messages identification
• Duplicate detection on content
• Near dup detection
• Entity extraction
• Emotional intelligence scoring:
  • rationalization,
  • opportunity,
  • intent,
  • pressure,
  • roundabout style,
  • positivity,
  • negativity
• Concepts and concept weighting
• Clustering for cluster wheel
• Document vectors and metadata analysis used for Supervised Learning classifiers

What triggers an AI Doc Sync job to start?

AI Document Sync jobs will be triggered automatically when the indexing of new document text and metadata completes, for example, when processed data is exported to Review. When the basic indexing finishes, the AI sync job is created. There is a 60-minutes delay before the AI job begins. The delay clock starts at the conclusion of the last to finish of the following (these are on a project level):

• Index Jobs
• Deletion Jobs
• Other AI Sync jobs

If there are any issues with an AI sync job, the process can be manually restarted. See below for more details on this topic.

What triggers an AI Doc Sync job to start?

AI Document Sync jobs will be triggered automatically when the indexing of new document text completes, specifically when OCR/Loaded and/or Extracted text sets are indexed. Metadata indexing does not trigger the creation of an AI Sync job. For example, when processed data is exported to Review and the basic indexing of OCR/Loaded and/or Extracted text sets finishes, an AI Sync job is created. There is a 60-minute delay before the AI sync job begins. The delay clock starts at the conclusion of the last to finish of the following (these are on a project level):

• Index Jobs

• Deletion Jobs

• Other AI Sync jobs

If there are any issues with an AI sync job, the process can be manually restarted. See below for more details on this topic.

How to check on the status of an AI sync job.

AI syncing consists of two main phases which can be observed on the Jobs page. Each sync begins with the AI Document Sync job. That process runs through many stages. When it is completed, the data is then loaded into the Review system and goes through Indexing to make it available for use. To check on the job status:

1. Open the Jobs page from the Projects dashboard, the menu on the left of the Review page, or from the flyout menu in Admin. Access to this is controlled by permissions, so not all users will have access.
   
2. Go to AI Document Sync The job will be shown on this screen with details on status, document counts (Total, Added, Updated, Deleted), time, and current status details. If the Status column is showing a green gear icon, the job is running. The status details display some information on the job while it runs. (Not all stages are counted so the step numbering will appear off. That is expected.)
   
3.  When the AI Sync completes (status shows green check) the next place to check is the Index page to view current indexing jobs. Click View All Jobs at the top of the Index page to see the analytics index jobs. Note there will be a short delay in these jobs starting. There are two jobs that index all of the AI analytics content - 'Brainspace DB Ingestion' and 'Brainspace Integrated Metadata Indexing'. When these are completed, the analytics data is available in Review. 
   
4. In addition to these jobs that you can view, there are some processes that are not displayed on the Jobs page. These are related to the Reveal AI ingestion and do not impact any of the visualizations or search features that use the AI data. Only Supervised Learning classifiers may be impacted. One example would be seen if you try to create a classifier using an AI Model. The classifier status will be Busy with the detail “Waiting for Vectors”.

Indicators of a failed AI Sync

There are several ways to verify if a sync job has failed. Below will list these and the next section of this document will describe how to resolve these issues.

• From the Jobs screen, check the AI Document Sync and Index pages to verify if the job was successful or failed. A failed job will have a red yield icon. The details will give an indication of what stage failed. This could be a system error or an ingestion error.
  
• If all visualizations on the Dashboard that use AI data display a message that “Analytics not yet available” then the AI job is still running or has failed. The same is true if the Cluster Wheel is missing and Concept Search says it is not available.
  

• The Documents candy bar visualization on the Dashboard that shows counts of originals, dupes, near dupes, and not analyzed is missing OR the document counts do not match with the entire dataset counts. This can be caused by the entire AI sync job or the Brainspace build failing.
  

• When some but not all of the Dashboard visualizations are available, this may indicate that some of the Indexing jobs have failed or are still running. Examples are:
  • Entities facet on the Dashboard is empty.
  • Emotional Intelligence Scores are missing, or all documents are unscored.
    
  • Senders + Recipients facet on Dashboard is empty.
  • Domain facet is empty.
  • Cluster Wheel is missing, has 1 big cluster, or has no clusters.
  • Cannot search for the any of the BD or BD EMT fields such as BDID, BD EMT IsUnique.
• Supervised Learning Classifiers are stuck in the “Busy – Waiting for Vectors” status.

Re-running the AI Sync

If it has been determined that an AI Document Sync job has failed, any of the Indexing jobs run post AI sync have not completed, and/or Classifiers are stuck in “Busy – Waiting for Vectors”, the admin can re-run the job. There are various options available in the AI Document Sync job to restart at the stage where the issues occur.

Restart options for an AI sync job can be found on the AI Document Sync Jobs page. Click the down arrow next to the status icon. You will see several choices, as displayed below. Select the appropriate starting place based on the issues that have been observed.

• Start Full – This will restart the entire AI sync process from the beginning, including ingesting the text and metadata. Run this option if the job failed completely and all information is “missing”. This can also be run in lieu of the separate stages. Keep in mind that this option will take longer to complete than a specific step.
• Start from Brainspace Build– Skips the ingestion into Brainspace and processes from point of failure. When complete, it then runs Reveal AI sync and Brainspace Reports indexing. Run this if AI job failed during Brainspace stages as seen in the job status.
• Start from Reveal AI– Skips ahead to run only the AI sync. It will also rerun the Reports indexing. Run this if Supervised Learning errors occur or if the job status message says RAI or NexLP failed.
• Start from Brainspace Reports– If any of the Index jobs of the Brainspace reports and overlay shows a failed status, run this job. Also, run if any of the Dashboard visualizations are missing. This option will index the existing AI sync reports without recreating them. While the other restart options include this phase, it is faster to run this when the problem is with the reports.
• Delete – Remove a sync job from the list. Use with care as a new sync job will only start when new data is indexed or full index is rebuilt, thus triggering the automatic sync.

When to contact Support.

When an AI Document Sync jobs fails, it is recommended that you check the Jobs to see where it failed and check the user interface for the clues as stated above. If it can be determined where the failure occurred, restart the AI sync at that point. Optionally, the Full process can also be run to redo the entire process. If there are continued failures, it is recommended that a support ticket be opened. There may be other factors that Reveal Support can pull from the logs to help troubleshoot.

Last Updated 7/15/2024