BDD Program - Rapid Pilot Project Creation

Document Purpose

The purpose of this document is to provide the instructions to create the BDD Program "Rapid Pilot Project". This project consists of an Enron subset, and a handful of documents to showcase our Review Accelerators (Translation, Image Labeling, and Transcription) totaling ~40K documents. This allows us to have a project for demo purposes.

The next step following this project creation is to view the recording of the standard Reveal Demo and access the demo script

Technical Documentation

To access all technical documentation within the Reveal Discovery Platform, please go to our Online Help Portal. 

High Level Process Overview

Below provides the high-level illustration of this entire process.

Access to Load Machine

To access a Load Machine, you can follow the high-level steps below.

1. https://reveal.teleport.sh/web/login

2. Use your Reveal Global ID to login

3. Find/click on the icon for your Load Machine by searching for the MSA

   1. *Please take note of the MSA or 8-digit number in the name of the machine. This is the MSA number for the pilot.

4. If you cannot find or access your Load Machine, please email support@revealdata.com and ask for access.

Stage Test Data

*The test data already exists if doing a Rapid Pilot POC.

There are two steps to stage the test data.

1. Download the test data to the Load Machine.

   1. On the Pilot’s Load Machine, use the following link to download the test data.

   2. https://revealdata-public-test-data.s3.us-east-1.amazonaws.com/20240303_RapidPilot.zip

      1. This link will automatically create a download in the browser when copied & pasted.

2. Copy/Unzip the data to the root of the Z:\ drive on the Load Machine so it looks like the below.

Project Creation

Reveal Review Manager is used to administer your instance of Reveal Review. In Review Manager, you can do any number of functions associated with the administration of Review such as create databases, manage users, load large amounts of data, set analytics and more. To see more information about this, please see Reveal Review Manager Overview.

Create Company:

*The Reveal Company already exists if doing a Rapid Pilot POC. If doing a custom Pilot POC, complete the following:

A Company is the highest level of organization in our permission structure. Please create a Company named Reveal.

1. Open Review Manager on the Load Machine

2. Expand Instance Setup

3. Click Companies

4. Click the Add New tab

5. Enter Reveal (or preferred name) as the Company name, and click Insert

Create Client

1. In Review Manager on the Load Machine, expand Instance Setup and click Clients

2. Click the Add New tab, enter the HubSpot POC ticket number as the Client name, enter a Client Number (typically a number, recommended to use date created in YYYYMMDD format), associate the Client with the Reveal company, and Click Insert.

Create Rapid Pilot (RP) Enron Project

After creating or using a previously created Company and Client, follow the steps below to create a new Rapid Pilot.

1. In Review Manager on the Load Machine, expand Instance Setup and select Projects

2. Click the New tab

3. Enter the Project Name (use ‘RP_’ as the prefix, the client's MSA_Enron as the project name, i.e. RP_81010001_Enron), enter a Project ID (typically a number, recommended to use date created in YYYYMMDD format), associate the Company/Client, choose the Reveal InControl Standard Template, and UTC for Time Zone. (NOTE: If you do not see your newly created “Client Name”, close the open tabs at the top of the screen by right clicking)

4. Click Create Case (Will take a couple of minutes to create)

User Setup

After creating the project, there are a few things that must be done prior to importing the data.

Create Review Account and Add to Rapid Pilot Project

Create a Review account for yourself and make your user a Company Admin as detailed below.

1. In the Review Manager on the Load Machine, expand Project Setup and click Users

2. Click “Add” and find your username. Add your user to the newly created Project as an Administrator

3. Click Create New, fill in the Create a new User ID form, and click Create after you've entered in the requisite information for all of the POC users.

4. Change Group to Administrators

Company Admin Rights

A user must have Company Admin rights to create users or a project within Review. To make a user a Company Admin follow the steps below.

1. Open Review Manager on the Load Machine

2. Expand Instance Setup, and select Companies

3. Ensure the Company created associated with Rapid Pilot is the one selected

4. Click the Users By Company tab

5. Click Add User, and choose your user

6. After adding the user select the checkbox Is Admin.

Checking for Company Admin Rights in Review

If a user does not have access to the Load Machine, but wants to check for Company Admin rights, the user can check within Review.

1. Log into Review

   1. You can get the URL to Review to any environment in Review Manager by expanding Advanced Options, click System Settings, and in the Setting column type 'indexserverserviceclient'. When copying this URL result, exclude '/review_indexserver' from the result.

2. Click Admin

3. Click the Menu button in the upper left-hand corner and look for Company Admin.

Please note if Company Admin is not visible, a user with access to the Load Machine will need to grant Company Admin rights via the steps above.

Import Data

We will import data into the project. This requires importing the DAT file into the system and creating indexes. To do this, follow the steps below. Please note the project in the screen shot is for illustration purposes only, and you should be choosing your project you created.

Import Mappings

A new field mapping will also need to be created to load the data.

1. Open Review Manager on the Load Machine, Expand Project Setup and Click Import Mappings

2. Browse to the following DAT file

   1. Z:\20240303_RapidPilot\20240303_RapidPilot_loadfile.dat

3. Click Match All Fields.  

4. Sort by the field Table Name, click in one of the fields that say -- Select -- and find/remap only the fields that show here as they did not map properly (there will not be that many). The fields that appear under the drop-down are the fields to find under table name. Remap each of these fields.  When this is complete, there will be no more fields remaining in the drop down. 

   1. APPTBEGDATE

   2. APPTENDDATE

   3. DATE_OTHER

   4. FILESIZE

   5. LAST_ACCESS_DATE

   6. MASTERDATE

   7. SENT_DATE

   8. TIME_OTHER

5. After mapping the unmapped fields, save this mapping as the Rapid Pilot Mapping

   1. Continue without mapping Conversation_ID if warning comes up. Please note that all fields will not map. You will only map the fields that show via step 6 above.

Documents Import

To import the DAT file, follow the steps below.

1. Open Review Manager on the Load Machine

2. Expand Import

3. Click Documents

4. Choose the Rapid Pilot project

5. Choose the Rapid Pilot Mapping

6. Import Data File: Z:\20240303_RapidPilot\20240303_RapidPilot_loadfile.dat

   1. Click Import Data

   2. Click OK

      

   3. This will load 40,839 files. (Approximately 1-2 minutes)

 

Create Indexes

After importing the data, you need to index the data. Please only do this after Store Pending = 0 as seen below. As part of this process, we grab the native and text files from the load file, and move them to S3 in AWS, or the “store”.

Please note the project in the screen shot is for illustration purposes only, and you should be choosing your project you created.

1. Open Review Manager on the Load Machine, Expand Create, Click Indexes

1. Choose the Rapid Pilot project

2. Select the index

3. Click Index/Re-Index

4. Click Native/Html, OCR / Loaded, Document_Metadata, and click OK (Approximately ~1 hour for all indexes to complete)

If a pop up regarding another indexing job comes up, click yes and continue.

Create Document Folders

After kicking off indexing, run the following process in the Create section of Review Manager.

1. Document Folders

   1. Choose the Rapid Pilot project

   2. Select the data set

   3. Change Document Folder Field to RELATIVEPATHPARENT

   4. Click Run (Approximately 3 minutes)

1. Alternatively, you can run document folders from the Project Admin settings on the Review site. Go to the Project → Project Admin → Build Document Folders → Follow the steps above in picking the data set and folder field.

   

    

Monitor Progress

1. You can monitor the indexing progress in the Review Manager, or you can go to Menu -> Jobs -> Index, select View All Jobs, and you will see the indexing process of the import. Make sure the counts in your project match the counts in the screen shot below except for Total Errors.

2. Make sure the data loaded properly into AI by going to Menu -> Jobs -> AI Document Sync. If this shows complete

3. Everything will be complete when the Candy Bar displays counts on the Dashboard, and that the index Brainspace connector overlay completed as well.  

It can take upwards of 20 minutes for the Candy Bar and/or the Brainspace connector overlay to display after the AI Document Sync has completed. 

Run Final Processes

These final processes will showcase the following key features of Reveal during the demo:

• Key Documents

• Review Accelerators (i.e., Image Label, Translation, and Transcription)

• Enable AI

• Field Profile

The project in the screen shots is for illustration purposes only, and you should be running these processes in the project you created. 

 

Key Documents

For demo purposes, the following documents highlight key differentiators in Reveal. To get to these quickly, it is recommended to add these to a Work Folder.  

1. Login to your project via your browser in Review

2. Next to Work Folders on top left, Click the ellipsis

   

3. Click Add Folder -> Name Folder "Key Docs" -> Click to Grid View

4. Click into Search Toolbar at top right, Click the ellipsis

5. Click +Add -> Type Begin Number -> Click Begin Number metadata field

6. Copy and paste the following Begin Numbers into the pop out window: 

   1. RP-00003880
      RP-00007310
      RP-00007374
      RP-00007396
      RP-00007410
      RP-00007448

7. Click Add to Search -> Click Search -> Results should look like documents below:

8. Click Update -> Action to Take -> Check Folders -> Click dropdown to Work Folders -> Check off Key Docs -> Click Submit -> Confirm

The work folder and documents selected can be selected based on preference for demo purposes - just a suggested set of documents to highlight Review Enhancers within the Reveal Platform.

Review Accelerators - Label

Run Image Label on the following images for demo purposes.

1. Login to your project via your browser in Review

2. Select the Documents Folders Documents → 02 → Group Share → Images, and this should return 49 files

   1. Note: If the top level folder does not return 49 files, click the dropdown arrow and initiate the process outlined below on each sub-folder.  

3. Click the Label button on toolbar

4. Click SUBMIT

5. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

Review Accelerators - Translate

There are two sets of documents for translation purposes of Japanese and Korean. Below illustrates how to translate the Japanese files. The process should be repeated for the Korean files as well.

1. Login to your project via your browser in Review.

2. Select the Documents Folder Documents → 02 → Group Share → Japanese and this should return 30 files

   1. Note: If the top level folder does not return 30 files, click the dropdown arrow and initiate the process outlined below on each sub-folder.  

3. Click the Translate button

4. Next to Destination Text Set click the + Create new button, enter the following items, and click Submit.

    

   

5. Back on the Translate Documents settings, choose the following options.

   

   1. Note: If the Destination Text Set that was just created does not show up in the list of text sets to choose from, logout/login, and the text set will be available. 

6. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

Repeat this process for the Korean files (Documents Folder Documents -> Dropbox -> Group Share -> Foreign Languages -> Korean), but translate from Korean to English. Please add the Korean files to the same Text Set of Translation, and do not create a new Text Set.

 

Review Accelerators - Transcribe

There is 1 deposition video to transcribe.

1. Login to your project via your browser in Review.

2. Select the Documents Folder Documents → 02 → Group Share → Videos, and this should return 1 file

3. Click the Transcribe, and click TRANSCRIBE (leaving all default settings)

4. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

Review Accelerators - Show Transcription

After the deposition video completes, turn on the transcription to show it during a demo.

1. Login to your project via your browser in Review.

2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Videos, and this should return 1 file

3. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

4. Launch this file in Review by clicking on the row.

5. Click the CC Show Transcript button , and this will show the text next to the video when it is played. This setting will stick for all future demos of this file.

While the Review Enhancers are executing, several additional tasks can be done to get the project ready for demo purposes which are outlined below.

Enable AI - Add AI Model

Apply an AI Model from the Model Library for demo purposes. 

1. Login to your project via your browser in Review

2. Click into Project Admin -> Tags

3. Click Add Tags and Choices 

4. Enter and select the following information for the tag:

    

   
   

    

5. Return to your main Project screen -> Click Supervised Learning

6. Click the gear icon by the AI Tag created (Junk)

7. Scroll through the AI Library Models and add the "Out of Office" AI Model

8. Click Run Full Process -> Click Save (at bottom)

9. Click the Review Icon on left panel, return to dashboard. 

The project's documents will now be scored with this AI model as part of the current Classifier which will take some time to run the process before it displays on the dashboard.

You can replicate these steps to create another AI Model for demo purposes or even  combine two similar models from the library to "pack and stack" into one uber AI Model to further leverage the AI Model Library.

For more information on applying or leveraging AI Models, please see or help documentation or blog posts. 

Enable AI - AI Responsiveness Mutually Exclusive Tag

Update the existing Responsiveness tag to enable an AI model for the tag.

1. Login to your project via your browser in Review

2. Click into Project Admin -> Tags

3. Click edit (pencil icon) next to Responsiveness, and enable AI

4. Select "Edit Access" next to Prediction AI Access and select Original Administrators to have access to view AI Prediction Scores 

Enable AI - AI Issues Multi-Select Tag

Create a new multi-select tag named Issues, and enable an AI model for each choice within the tag.

1. Click Add Tags and Choices 

2. Label Tag Choice "Issues"

3. Chose Multi-Select Tag 

4. Add the following issues separately: Chewco; Jedi; Raptor

5. Select Prediction Enabled for each tag

6. Select "Edit Access" next to Prediction AI Access and select Original Administrators to have access to view AI Prediction Scores

Enable AI - Finalize Tag Profile

1. Move the issues over to the Standard First Level Review Tag Profile that contains solely those tags or other tags relevant to typical workflow 

2. Click the Settings button, and make sure the Original Administrators have access to the Field Profile.

Enable AI - Populate Scores for AI Tags 

To get scores generated in the AI Tags, conduct the searches below, and bulk tag the results.

1. Login to your project via your browser in Review

2. Click into Grid View

3. In the top search toolbar enter in search for keyword Chewco

    

   

4. Click Update from Mass Action Toolbar

5. Click Action to Take ->  Select "Include Tags" -> Select Responsive -> Select Chewco -> Click Submit -> Click Confirm in pop up window
   
   

   

6. The AI Tagging needs both positive and negative examples to populate AI Scores accurately. To tag the negative documents follow the steps below.

7. In the top search bar, search for the word Chewco -> Add families from the tree expander -> search for Chewco again -> Change the boolean operator from OR to AND -> change boolean operator IS to IS NOT -> Click Search

    

   
   

   

8. With the populated results, use the Mass Actions Toolbar again. 

9. Click Action to Take ->  Select "Include Tags" -> Select Non-Responsive -> Click Submit -> Click Confirm in pop up window

10. Repeat these same actions for the issues tags LJM, JEDI, and RAPTOR (e.g. search for LJM instead, mass tag all docs from search result with respective tag) 

If a window appears suggesting the action may take a while, monitor in Jobs; however bulk tagging will process fairly quickly. 

AI Scores from these mass tagging actions will update after several minutes after all Issues are mass tagged and synced with AI tagging engine. 

When this completes (takes roughly 5 - 15 minutes), 5 new models will populate on the Dashboard in the widget Documents by Predictive Scores with the results.

Field Profile

By default, field profiles are not assigned to roles out of the box. This makes the Default Field Profile inaccessible for all users. To fix this, follow the below steps, and you can use the Default field profile. 

1. Login to your project via your browser in Review

2. Select Admin 

3. Click Fields -> select Default -> click Edit

4. Select Original Administrators -> choose Fields -> click UPDATE

    

   

   1. Recommended Fields:

      1. Begin Number

      2. Begin Number Attach

      3. Master Date time

      4. Custodian

      5. Extension

      6. App Name

      7. Email Subject

      8. From 

      9. To

      10. CC

      11. BCC

      12. Responsiveness

      13. Issues

      14. Foreign Languages

5. Switch to Grid, and choose Default if not selected

There are many other fields that could be added. These are just a starter kit, which can be expanded

 

Name Normalization

Merge different aliases for the top 5 custodians in the data.

1. Login to your project via your browser in Review

2. Select the Communications view

3. Select the Data button

   

4. For each of the top 5 custodians, do the following:

   1. Select Communicators

      
      

      

   2. Search for the Custodian

      
      

      

   3. Select all associated names for that custodian → Hit the merge button

      
      

      

   4. Select the name you want all the aliases to merge into