To import data via SFTP in the background on a regular basis, contact with our helpdesk is required, and a subscription to our Backgrounding addon is also required.
Configure your SFTP credentials in the import wizard by selecting SFTP as the data source.
Provide your credentials and path as shown below. MetaLocator will provide SFTP credentials to you. You should upload your file to this SFTP repository on a daily basis. The file should be uploaded as a UTF-8 encoded comma-separated value file (CSV). The file should follow a naming convention as follows:
locator-YYYYMMDD-HHMM.csv
This format allows for easy disambiguation of multiple files and allows MetaLocator to determine the most recently uploaded file even when file timestamps have been changed.
The "Local file name" setting determines the name of the file used to look up the import options and mapping rules. This is the file name the remote FTP file is "saved as" after the SFTP transfer is complete.
Settings are as follows, and are provided by the helpdesk.
SFTP Username: The username to login to sftp.
SFTP Password: The password to login to SFTP. Most MetaLocator implementations are key-based logins, so this can typically be left empty.
SFTP Host: Typically
securetransfer.metalocator.com.SFTP Remote Path: The full path to the file on the SFTP server. Can contain wildcards. E.g.
/home/<username>/locator-*.csvLocal file name: The name of the file to be downloaded. This must correspond to established mapping from a previous import of the same name. If specified, the local file name should be obscure, so that other imports do not inadvertently disrupt the mapping rules associated with the filename. E.g.
12639-locator.csvEmail Notify List: A semi-colon separated list of email addresses to CC upon completion of the SFTP import task.
RSA Private Key (PEM Format): This is the private key used to connect to MetaLocator SFTP.
Other job options include:
sftp_latest_only: 1 or 0. Determines the latest file to download based on the file's modified time. Used only in combination with a wildcard specification in the SFTP Remote Path setting.
sftp_skip_data_combinator: 1 or 0. If set to 1, bypasses any custom import plugins for this job.
sftp_once_only: If set to 1, and the sftp_remote_path contains a wildcard and the is not set, the system will not re-process the same file if downloaded twice as determined by a filename match. Do not enable this option if the remote filename is static, or the local filename is set.
sftp_sync: No longer used.
sftp_delete_after_download: 1 or 0. If set to 1, MetaLocator will delete the file from the remote SFTP after a successful download. If multiple files are found that match the SFTP Remote Path, MetaLocator will process the latest, according to the system modified (mtime) and delete the rest of the files that match the SFTP Remote Path expression. This ensures that if multiple files are present, the job only processes the latest file and does not work its way through progressively older versions of the file.
Setting up a SFTP Job
The steps to setting up an SFTP job are as follows:
Before beginning:
The MetaLocator help desk creates and provides the SFTP credentials upon request.
The customer provides an IP address or range to MetaLocator to whitelist through the SFTP firewall.
The customer uploads a file to the SFTP server in the required format on a regular schedule.
The file is verified as ready for direct import
Setting up the import:
The user running the job must be in the Backgrounding group.
Download and import the file manually via Data > Import > CSV upload to establish the column mapping and import settings. The filename used should be the "Local file name" as chosen above in the SFTP job settings.
Import using Update Existing and Insert New or similar repeatable options that will not create duplicates as the job is repeated.
Establish the SFTP job under Data > Import > SFTP as described above. Ensure the "Keep in Sync" checkbox is checked so that a job is created.
Configure the resulting SFTP background job as described above.
Run and schedule the SFTP job. Forcing it to verify that the job runs successfully as evidenced by an email and account notification.
A properly configured import job is shown below:
In the above example, the user has already imported a file called 12639-locator.csv manually, so that when this job runs, the mapping is loaded based on the settings chosen during the import.
Before beginning, make sure you can download the most recently uploaded CSV file as described here.
The most recent 30 files will be retained for up to a maximum age of 120 days. The SFTP server should not be used as a backup facility or for any other purpose than the transmission of data for import into MetaLocator.