SFTP

Description

Create an API that can trigger SFTP actions.
The SFTP API can be used to pull and push files to the API server. This API makes the API server a centralized location for file transfer management

UI

Access the DataTrucker URL via a browser

Create a credentials

1. Go to Credentials in the sidebar > SSH Credentials
2. Scroll down to Credential Management
3. Fill the form to create the connection Credential name – an arbitrary value to identify the credential object
4. The hostname of the target server to SSH. Note: SSH executes from the API server
5. SSH Username
6. SSH password or location of the ssh private key on the API server
7. Password is Privatekey, is to identify if step 6 is a password or file name ( folder hosting all your keys are defined in the server setup config files )
8. SSH Port
9. Click on Create

Create a SFTP API

1. Go to File API in the Sidebar > SFTP APIs.
2. Fill the form to create the API
   1. Resource Name: an arbitrary name to identify the resource
   2. The Credential linked to the resources.
   3. Method: The type of REST Call
   4. SFTP Flow/location: download from a remote source to source location or upload to remote source from the source location
   5. Source Type: Indicates if the SFTP should move a file or a folder
   6. Target: location on the remote machine
   7. Validation Regex of Input values, input sanitization before querying
      1. Examples provided in the UI when you try to create a new API

Query the resource you created

################################ query a trigger SFTP action#############################
URL: /api/v1/jobs/<resource name>TYPE: <method defined>HEADER: Authorization: "Bearer <JWT Token>"
Response: 200 OK{   jsondata....}
#############################Note: Irrespective of the method chosen, this API will not parse the body

As a CRD in Openshift / Kubernetes

For credentials use the the API below to the management end point

---apiVersion: datatrucker.datatrucker.io/v1kind: DatatruckerFlowmetadata:  name: datatruckerflow-samplespec:  Resources:    requests:      memory: "256Mi"      cpu: "250m"    limits:      memory: "256Mi"      cpu: "500m"  JobDefinitions:  - resourcename: fileupload3    name: fileupload3    tenant: Admin    type: File-SFTP    restmethod: POST    credentialname: sftpconnect1    source: download    target_path: "/home/trucker/dt2/datatrucker/.gitignore_test"    source_type: file    source_path: "/upload/.gitignore"  Keys:    configmap: placeholder  Scripts:    configmap: placeholder  Type: Job  DatatruckerConfig: datatruckerconfig-sample  Replicas: 1  API:    name: api    Image:      repository: docker.io      imageName: datatruckerio/datatrucker-api      tagName: latest
  

API

Create a credential via REST

############################### create a SSH credential#############################
URL: /api/v1/credentials/TYPE: POSTHEADER: Authorization: "Bearer <JWT Token>"BODY (JSON): {        "credentialname": "sftpconnect1",        "type": "File-SFTP",        "hostname": "localhost",        "username": "foo",        "password": "pass",        "port": 22      }

Response: 201 OK
################################## create a SSH Private credential################################
URL: /api/v1/credentials/TYPE: POSTHEADER: Authorization: "Bearer <JWT Token>"BODY (JSON): {        "credentialname": "sftpconnect2",        "type": "File-SFTP",        "hostname": "localhost",        "username": "foo",        "port": 22,        "password": "id_rsa",          // folder defined in resource.config.json        "passwordisPrivateKey": true}
Response: 200 OK

Create the API via REST

############################### create a  download resource#############################
URL: /api/v1/resourcesTYPE: POSTHEADER: Authorization: "Bearer <JWT Token>"BODY (JSON): {        "resourcename": "fileupload3",        "type": "File-SFTP",        "restmethod": "POST",        "credentialname": "sftpconnect1",        "source": "download",        "target_path": "/home/trucker/dt2/datatrucker/.gitignore_test",        "source_type": "file",        "source_path": "/upload/.gitignore"}
Response: 201 OK

############################### create a upload resource#############################

URL: /api/v1/resourcesTYPE: POSTHEADER: Authorization: "Bearer <JWT Token>"BODY (JSON): {        "resourcename": "fileupload",        "type": "File-SFTP",        "restmethod": "POST",        "credentialname": "sftpconnect1",        "source": "upload",        "source_path": "/home/trucker/dt2/datatrucker/.gitignore",        "source_type": "file",        "target_path": "/upload/.gitignore"}

Response: 201 OK