SQL Loader - Updates to account causes "next line" and "space" related issues in the file

Problem:
I’m requesting access for an account on SQL Loader source from ISC. The SQL loader source is connected to a text file on a remote server. For the first request where the account is created, the file looks perfect where a new entry is added with the user record. However, when requested additional access or make changes to the account from SailPoint, below issues are being identified:

  1. The connector writes the account data in the same line in continuation with the previous record.
  2. The account updates cause the connector to write extra spaces into the file leading to the file having unnecessary spaces.

I already went through the below documentation and discuss post, but they were not so useful:

Can someone suggest what could be a problem here? This looks to be a common issue with the JDBC Text driver (com.hxtt.sql.text.TextDriver).

Appreciate a solution for this if someone has worked on the same.

Tagging for visibility since I’ve noticed you talking about similar issue on different posts - @patrickboston @RobBuskens @varshini303 @Kyle_R @agutschow @Yadnyesh_Kulkarni @Paul_Meyer @varanjo

Thanks,
Arshad.

1 Like

@Arshad

Spaces are added to CSV and text files during provisioning

Resolution – To avoid adding spaces into the CSV or text files while performing provisioning operations, add the following query in the corresponding rule:

String query1 ="pack table table_name";

stmt.execute(query1);

Hey @RobBuskens Thanks for the response. I saw this statement in the troubleshooting section of the connector document as well.

Can you help me understand where exactly this needs to be added in the JDBC Provision Rule? I tried adding it but it kept saying “Unspecified Connector Failure” so I think I’m missing the information on when and where this snippet must be placed in the rule.

Probably a sample rule where you have these ‘pack table’ related statements might help us a lot!

Thanks,
Arshad.

Hi @Arshad ,
Can you check below link ? The have all the steps how to write the code along with attaching it to the connector.

@RAKRHEEM, I know exactly how to write a JDBC Provision rule and the rule code that I’ve written works perfectly fine for create account/add entitlement/remove entitlement/disable accounts.

To re-iterate my question, I wanted to see how the PACK TABLE <table_name> related code snippet is incorporated into the rule because once I have statement.executeUpdate() in my code for provisioning related operations and I place the snippet there, the rule simply doesn’t work. Versus when I don’t have PACK TABLE <table_name> in my rule, all the use cases work fine but the extra spaces get added into the text file in the remote server.

Please try executing pack table {table_name} query in provisioning rule.

placement. can vary by use case. It’s only required after updates based on my experience.

After you’ve executed update statements (1 or more) but before you return from the rule, that’s where you put the pack table code.

I recommend putting it once at the end of all updates.

String packTableCommand = "pack table {tableName}";
PreparedStatement statement = connection.prepareStatement(packTableCommand);
statement.executeUpdate();

Where you’ll replace {tableName} with the table name to update.

if your getting an error, share it and we’ll see what’s going on.

As mentioned above, tried this and got Unspecified Connector Failure

Sure Rob, will try this and see how it goes. I tried couple of places where I placed this code snippet inside a method call where the account updates were being done and it said “Unspecified Connector Failure”.

Let me try something different and keep you posted.

Thanks,
Arshad.

Are you setting the provisioning result?

    try {
          // Do stuff
        } catch(Exception e) {
          result.addError(e.getMessage());
          result.setStatus(ProvisioningResult.STATUS_FAILED);   
         } finally {
             if(stmt != null)
             stmt.close();
      }
      return result;

this way the error may be specified.

@RobBuskens, Below is the blueprint of my rule. I have defined a method executePackQuery to handle the pack statements related execution. When I run this rule, there is no issue with the provisioning to the file. However, the spaces still do exist in the file after provisioning. Due to these spaces, it is causing me issues during deprovisioning where I want to delete the user records from the file, the connector is unable to determine that the records do exist towards the end of the file after the spaces, resulting in incorrect deprovisioning actions on the file.

// source code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Types;
import java.util.List;
import sailpoint.api.SailPointContext;
import sailpoint.connector.JDBCConnector;
import sailpoint.object.Application;
import sailpoint.object.ProvisioningPlan;
import sailpoint.object.ProvisioningPlan.AccountRequest;
import sailpoint.object.ProvisioningPlan.AttributeRequest;
import sailpoint.object.ProvisioningPlan.PermissionRequest;
import sailpoint.object.ProvisioningResult;
import sailpoint.object.Schema;

//Global constant
Identity identity = null;
AttributeRequest attrReq = null; 

public void executePackQuery(Connection connection) {
    PreparedStatement packStmt = null;
    try {
        String packQuery = "pack table <my table name here>";
        packStmt = connection.prepareStatement(packQuery);
        packStmt.execute();
    } catch (SQLException e) {
        log.error("Error executing pack query: " + e.getMessage());
    } finally {
        if(packStmt != null) {
            try {
                packStmt.close();
            } catch (SQLException e) {
                log.debug("Error closing pack statement: " + e.getMessage());
            }
        }
    }
}

public boolean myMethod (Connection connection, AccountRequest account) { 
	PreparedStatement statement = null; 
	String QUERY = "my SQL query";
	try { 
		if(entitlement.size() > 1){ 
			for(String role : entitlement){ 
				statement = connection.prepareStatement(QUERY); 
				statement.setString (1, <attr>); 
				statement.setString ( 2, <attr>); 
				statement.setString ( 3, <attr>); 
 
				//Execute the statement 
				statement.executeUpdate();
			}

            // Now pack the table to clean up extra spaces
            executePackQuery(connection);

			return true; 
		} 
		else if(entitlement.size() == 1) { 
			statement = connection.prepareStatement(QUERY); 
			statement.setString (1, <attr>); 
			statement.setString ( 2, <attr>); 
			statement.setString ( 3, <attr>); 
			//Execute the statement 
			statement.executeUpdate();

            // Now pack the table to clean up extra spaces
            executePackQuery(connection);

			return true; 
		}
	} catch (SQLException e ) { 
		
	} finally { 
		if(statement != null) { 
			try { 
				statement.close(); 
			} catch ( SQLException e ) {
				//
			} 
		} 
	}  
	return false; 
}
 
log.info( "Entered the Rule" ); 
ProvisioningResult result = new ProvisioningResult(); 
 
if ( plan != null ) { 
	List accountRequests = plan.getAccountRequests(); 
	if ( ( accountRequests != null ) && ( accountRequests.size() > 0 ) ) {  
		for ( AccountRequest account : accountRequests ) { 
			try {  
				if ( account.getOperation() != null && AccountRequest.Operation.Create.equals( account.getOperation() ) ) {    
					if (<myMethod call>) 
						result.setStatus( ProvisioningResult.STATUS_COMMITTED ); 
					else 
						result.setStatus( ProvisioningResult.STATUS_FAILED ); 
				} else if ( account.getOperation() != null && AccountRequest.Operation.Modify.equals( account.getOperation() ) ) {
                        if (<myMethod call>)
                            result.setStatus(ProvisioningResult.STATUS_COMMITTED);
                        else {
                            result.setStatus(ProvisioningResult.STATUS_FAILED);
                        } 
				}
				else { 
					//Unsupported Operation  
					result.setStatus( ProvisioningResult.STATUS_FAILED ); 
				} 
			} catch (Exception e) {  
				result.setStatus( ProvisioningResult.STATUS_FAILED ); 
				result.addError( e ); 
			} 
		} 
	} 
} else { 
  log.info("Plan is empty. No operations performed"); 
} 
log.info( "Exiting the Rule"); 
return result;

If you see the above code, I have placed executePackQuery(connection); to call the execute pack statement related method which is right before returning back to rule. Still does not help in cleaning up the garbage spaces in file.

Can you review and let me know of any issues here?

Thanks,
Arshad.

@Arshad I have been using Sailpoint IIQ and we have written below code in After Provisioning rule to handle blank lines that are added during update or delete operation.

import java.io.IOException;
  import java.nio.file.Files;
  import java.nio.file.Paths;
  import java.util.ArrayList;
  import java.util.List;
  import sailpoint.tools.Util;
  import org.apache.commons.logging.Log;
  import org.apache.commons.logging.LogFactory;

  //Code to remove emptylines caused in delete operation

  String filePath;
  String url = (String) application.getAttributes().get("url");

  String[] parts = url.split("////");

  if (parts.length >= 1) {
    filePath = parts[1]; 
  }

  try {
    if(Util.isNotNullOrEmpty(filePath))
    {
      log.debug("filePath"+filePath);
      List lines = Files.readAllLines(Paths.get(filePath));
      List nonEmptyLines = new ArrayList();

      for (String line : lines) {
        if (!line.trim().isEmpty()) {
          nonEmptyLines.add(line);
        }
      }

      Files.write(Paths.get(filePath), nonEmptyLines);
      log.debug("Empty lines removed successfully.");
    }  
  } catch (IOException e) {
    e.printStackTrace();
  }

  //End of code to remove empty line

Unfortunately, this after provision rule based solution isn’t feasible with SailPoint ISC.

I’ve opened a support ticket for the same and awaiting response.

Anyways, thanks to everyone for your inputs.

Regards,
Arshad.