Copyright (c) 2015-2017 Avere Systems, Inc.  All Rights Reserved.

The Avere virtual FXT (vFXT) Python library and vfxt.py command line utility
provide the ability to create, extend, destroy, start, and stop vFXT clusters in
all supported cloud environments.

Licensed under the Apache License, Version 2.0.

You can find the user guide here:
https://download.averesystems.com/software/avere_vfxt.py_usage_guide.pdf


0. Installation

  python setup.py install
    - or see python setup.py install --help options for customization

  python setup.py bdist_egg
    - manually install the resulting files
    cp vfxt.py dist/vFXT-0.1-py2.7.egg /some/path/

  python setup.py test
    - runs all unit tests
    - or run one suite: python setup.py test -s tests.GCE_test

  Authentication requirements specific for each service backend.

    - AWS: requires the access key/secret access key pair
    - GCE: requires a service account with the P12/JSON key file


1. vFXT Library


  - AWS Example:

    from vFXT.cluster import Cluster
    from vFXT.aws import Service

    aws = Service(region='fill me in', access_key='fill me in', subnet='subnet-f99a618e', secret_access_key='fill me in')

    cluster = Cluster.create(aws, 'r3.8xlarge', 'daily-test-20150309', 'adminpass')
    try:
      cluster.make_test_bucket(bucketname='daily-test-20150309-bucket', corefiler='daily-test-20150309')
      cluster.add_vserver('vserver')
      cluster.add_vserver_junction('vserver', 'daily-test-20150309')
    except Exception as e:
      cluster.destroy(remove_buckets=True)

    lg = aws.create_instance('c3.xlarge', 'lg1', 'ami-b9faad89', key_name='aws_ssh_key_name') # CentOS6-Avere-LG
    from vFXT.serviceInstance import ServiceInstance
    lg_instance = ServiceInstance(service=aws, instance=lg)
    # or via ServiceInstance which calls the backend .create_instance()
    lg = ServiceInstance.create(aws, 'c3.xlarge', 'lg1', 'ami-b9faad89')


  - GCE Example:

    from vFXT.cluster import Cluster
    from vFXT.gce import Service

    gce = Service(client_email='fill me in', key_file='path-to.json', zone='us-central1-b', project_id='fill me in', network_id='fill me in')
    cluster = Cluster.create(gce, 'n1-highmem-8', 'daily-test-20150309', 'adminpass')
    cluster.stop()
    cluster.destroy()

    lg = gce.create_instance('g1-small', 'lg1', 'centos-6-avere-v20150320')
    # or a public image like 'https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/backports-debian-7-wheezy-v20150710'
    with open('/Users/woodwardj/.ssh/id_dsa.pub','r') as f:
      sshpubkey=f.read()
      sshpubkey='woodwardj:{}'.format(sshpubkey)
    gce.set_metadata(lg, 'ssh-keys',sshpubkey)
    from vFXT.serviceInstance import ServiceInstance
    lg_instance = ServiceInstance(service=gce, instance=lg)
    # or via ServiceInstance which calls the backend .create_instance()
    lg = ServiceInstance.create(gce, 'g1.small', 'lg1', 'centos-6-avere-v20150320', metadata={'ssh-keys':sshpubkey}, tags=['avere-dev'])

    lg_instance.destroy()


  - To load an existing, running cluster:

    Cluster.load(service, mgmt_ip='fill me in', admin_password='fill me in')


  - To instantiate a cluster that may be offline:

    cluster = Cluster(gce, nodes=['xxx', 'xxx', 'xxx'], admin_password='xxx', mgmt_ip='xxx')
    if cluster.is_off():
      cluster.start()
    elif not cluster.is_on() and not cluster.is_off()
      # some nodes are offline, some are online
      cluster.status()


  - To serialize a cluster:

    import json
    json.dumps(cluster.export())

    cluster.export() emits
    {'nodes': [u'node-1', u'node-3', u'node-2'], 'admin_password': 'pass', 'mgmt_ip': '10.1.1.1'}

    # To recreate the object:
    cluster = Cluster(service, **{'nodes': [u'node-1', u'node-3', u'node-2'], 'admin_password': 'pass', 'mgmt_ip': '10.1.1.1'})

    # The same with your service object:
    service_data = service.export()
    service = Service(**service_data)



2. Example vfxt.py utility invocation:

  The first part of the invocations are the cloud-type and the authentication options.  Following those, the action and any
  related action options.

  # Use --key-file for json or p12 key file
  # --client-email is required when using a p12 PKCS#12 key file

  # GCE Create a cluster
  ./vfxt.py --cloud-type gce \
    --key-file=service-account.json \
    --project fine-volt-704 --zone us-central1-b --network gce1 \
    \
    --create                                \
    --image-id vfxt-4614                    \
    --admin-password adminpass              \
    --cluster-name woodwardj-$(date +"%s")  \
    --nodes 3                               \
    --gce-tag use-nat                       \
    --instance-type 'n1-highmem-8'          \
    --metadata 'Name:woodwardj-t010' --metadata 'Owner:woodwardj' --metadata 'Department:QA'

  # GCE Destroy a cluster
  ./vfxt.py --cloud-type gce \
    --key-file=service-account.json \
    --project fine-volt-704 --zone us-central1-b --network gce1 \
    \
    --destroy                         \
    --management-address 10.52.16.103 \
    --admin-password adminpass

  # GCE Upgrade
  ./vfxt.py --cloud-type gce \
    --key-file=service-account.json \
    --network gce1 \
    --project fine-volt-704 --zone us-central1-a \
    \
    --upgrade                         \
    --management-address 10.52.16.115 \
    --admin-password 'adminpass'     \
    --upgrade-url gs://woodwardj-storage/armada-hotel.1-54402c7.pkg

  # GCE Add nodes
  ./vfxt.py --cloud-type gce \
    --key-file=service-account.json \
    --network gce1 \
    --project fine-volt-704 --zone us-central1-a \
    \
    --add-nodes                       \
    --nodes 3                         \
    --management-address 10.52.16.115 \
    --admin-password 'adminpass'

  # Running on a GCE instance lets you use --on-instance to determine credentials
  ./vfxt.py --cloud-type gce --on-instance \
    --create                        \
    --image-id vfxt-4614            \
    --cluster-name test             \
    --admin-password adminpass      \
    --nodes 1                       \
    --instance-type 'n1-highmem-8'  \
    --metadata 'Owner:woodwardj' --metadata 'Department:QA'

  # GCE local-ssd (max 8 volumes, 375G SSD drive size)
  ./vfxt.py --cloud-type gce \
    --key-file=service-account.json \
    --network gce1 \
    --project fine-volt-704 --zone us-central1-a \
    \
    --create                            \
    --image-id vfxt-4614                \
    --cluster-name 'woodwardj-18505-4'  \
    --admin-password 'adminpass'        \
    --instance-type 'n1-highmem-8'      \
    --debug --local-ssd                 \
    --node-cache-size 375               \
    --volumes 4

  # GCE Create a cluster using CUSTOM interop credentials
  ./vfxt.py --cloud-type gce \
    --key-file=service-account.json \
    --project fine-volt-704 --zone us-central1-b --network gce1 \
    \
    --create                                \
    --image-id vfxt-4614                    \
    --admin-password adminpass              \
    --cluster-name woodwardj-$(date +"%s")  \
    --nodes 3                               \
    --s3-access-key X --s3-secret-key X     \
    --instance-type 'n1-highmem-8'          \
    --metadata 'Name:woodwardj-t010' --metadata 'Owner:woodwardj' --metadata 'Department:QA'

  # GCE interactive
  ./vfxt.py --cloud-type gce \
    --key-file=service-account.json \
    --project fine-volt-704 --zone us-central1-b --network gce1 \
    \
    --interact --debug


  # AWS Create
  ./vfxt.py --cloud-type aws --region us-west-2 --access-key 'X' \
    --secret-key 'X' --subnet subnet-f99a618e \
    --placement-group perf1 \
    \
    --create                                \
    --cluster-name woodwardj-$(date +"%s")  \
    --admin-password adminpass              \
    --nodes 3                               \
    --instance-type 'r3.2xlarge'            \
    --aws-tag 'Owner:woodwardj' --aws-tag 'Department:QA'

  # AWS Destroy
  ./vfxt.py --cloud-type aws --region us-west-2 --access-key 'X' \
    --secret-key 'X' --subnet subnet-f99a618e \
    --placement-group perf1 \
    \
    --destroy                         \
    --management-address 10.50.248.50 \
    --admin-password adminpass

  # AWS Add nodes
  ./vfxt.py --cloud-type aws --region us-west-2 --access-key 'X' \
    --secret-key 'X' --subnet subnet-f99a618e \
    --placement-group perf1 \
    \
    --add-nodes                       \
    --nodes 3                         \
    --management-address 10.50.248.50 \
    --admin-password adminpass


