Accessing in addition to Managing Multiple Clouds (Infrastructures) with CloudmeshJune 24 2

Accessing in addition to Managing Multiple Clouds (Infrastructures) with CloudmeshJune 24 2

Accessing in addition to Managing Multiple Clouds (Infrastructures) with CloudmeshJune 24 2

Gable, Gene, Contributing Editor has reference to this Academic Journal, PHwiki organized this Journal Accessing in addition to Managing Multiple Clouds (Infrastructures) with CloudmeshJune 24 2014BigSystem 2014 – Software-Defined Ecosystems at HPDC Vancouver Canada Gregor von LaszewskiFugang WangGeoffrey FoxIntroductionCloud computing has become an integral factor as long as managing infrastructure by research organizations in addition to industry. Public clouds: Amazon, Microsoft, Google, Rackspace, HP, in addition to others.Private clouds: set up by internal In as long as mation Technology (IT) departments in addition to made available as part of the general IT infrastructure“HPC Clouds”: Non hypervisor or high per as long as mance hypervisor based systems managed like cloudsCan we leverage all of themHow to deal with the frequent changing technologiesMinimal changes to users that only want to run an application!Use “Software Defined Infrastructure” in addition to “Software Defined Applications” FutureGrid has required this capability to build different software environments dynamically on it’s hardwareDescribe our Cloudmesh software approachCloudMesh ArchitectureTightly integrated software infrastructure toolkit to deliver a software-defined distributed system encompassing virtualized in addition to bare-metal infrastructure, networks, application, systems in addition to plat as long as m software with a unifying goal of providing Computing Testbeds as a Service (CTaaS).This system is termed Cloudmesh to symbolize:The creation of a tightly integrated mesh of services targeting multiple IaaS frameworksThe ability to federate a number of resources from academia in addition to industry. This includes existing FutureGrid infrastructure, Amazon Web Services, Azure, HP Cloud, Karlsruhe using several IaaS frameworks The creation of an environment in which it becomes easier to experiment with plat as long as ms in addition to software services while assisting with their deployment. The exposure of in as long as mation to guide the efficient utilization of resources.Cloudmesh exposes both hypervisor-based in addition to bare-metal provisioning to users.Access through comm in addition to line, API, in addition to Web interfaces.

Albright College PA

This Particular University is Related to this Particular Journal

Cloudmesh FunctionalityCloudmesh User Interface56

Cloudmesh Shell & bash & IPython 7Monitoring in addition to Metrics InterfaceService MonitoringEnergy/Temperature MonitoringMonitoring of ProvisioningIntegration with other ToolsNagios, Ganglia, Inca, FG Metrics, MonalyticsAccounting metrics8Operation in addition to Monitoring – HPC9

FutureGrid offersComputing Testbed as a ServiceFutureGrid usesTestbed-aaS ToolsProvisioningImage ManagementIaaS InteroperabilityNaaS, IaaS toolsExpt managementDynamic IaaS NaaS DevOps CloudMesh is a CTaaS tool that uses Dynamic Provisioning in addition to Image Management to provide custom environments as long as general target systemsInvolves (1) creating, (2) deploying, in addition to (3) provisioning of one or more images in a set of machines on dem in addition to 10Terms (Cloud & HPC)Public cloud: a service provider makes resources available to users over the public internet. This includes compute, storage, in addition to applications. FutureGrid offers a public cloud to its users.Private cloud: access to services may have additional restrictions. Restrictions could include a limited set of authorized users to the services offered or possible restrictions regarding exposing services on the public internet. FutureGrid offers the ability to set up private clouds as long as special projects. Examples include modified OpenStack deployments or reserved resources as long as classes. Hybrid cloud: a combination of public in addition to private clouds.Multi-cloud: access to a number of different clouds that may even use different IaaS or PaaS offerings.HPC service: a cloud service that allows the ability to run high per as long as mance computing jobs, as long as example on a compute cluster offering MPI.Provider consortium: is a (virtual) organization that integrates resources from multiple providers. We also can refer to such a consortium as a multi-cloud Grid.Terms (Cloud, HPC, Provisioning)Cloud Bursting:Use (external) clouds to provide additional (cloud) resources on dem in addition to Cloud ShiftingMove executing jobs from one cloud to anotherProvisioning: A process to install the operating system, data in addition to software to enable access to it.Rain: Process developed in FutureGrid that provisions the operating system, but allows the deployment in addition to configuration of useful in addition to complex services to be run on one or multiple machines in order to provide a service utilizing potentially distributed resources or services. It also contains the ability to re-provision servers in addition to services, that is, services may be suspended in addition to the resources used to run the service may be used by other services.Cloudmesh:Exp in addition to s Rain to general infrastructure

Background – FutureGridMany requirements originate from FutureGrid.This is a high per as long as mance in addition to grid testbed that allowed scientists to collaboratively develop in addition to test innovative approaches to parallel, grid, in addition to cloud computing. Users can deploy their own hardware in addition to software configurations on a public/private cloud, in addition to run their experiments. Provides an advanced framework to manage user in addition to project affiliation in addition to propagates this in as long as mation to a variety of subsystems constituting the FutureGrid service infrastructure. This includes operational services to deal with authentication, authorization in addition to accounting.Important features of FutureGrid:Metric framework that allows us to create usage reports from all of our IaaS frameworks. Developed from systems aimed at XSEDERepeatable experiments can be created with a number of tools including Cloudmesh. Provisioning of services in addition to images can be conducted by Rain.Multiple IaaS frameworks including OpenStack, Eucalyptus, in addition to Nimbus.Mixed operation model. a st in addition to ard production cloud that operates on-dem in addition to , but also a set of cloud instances that can be reserved as long as a particular project.FutureGrid coming to end but preserving CTaaS tools as CloudmeshFunctionality RequirementsProvide virtual machine in addition to bare-metal management in a multi-cloud environment with very different policies in addition to includingFutureGrid resources,External clouds from research partners, Public clouds,My own cloudProvide multi-cloud services in addition to deployments controlled by users & providerEnable raining ofOperating systems (bare-metal provisioning), ServicesPlat as long as msIaaSDeploy in addition to give access to Monitoring infrastructure across a multi-cloud environmentSupport management of reproducible experimentsUsability RequirementsProvide multiple interfaces includingcomm in addition to line tool in addition to comm in addition to shellWeb portal in addition to RESTful servicesPython APIDeliver a toolkit that isopen sourceExtensibleeasily deployabledocumented

Cloudmesh Definitions IProject: The research activity to be supported by Cloudmesh. A project has roles in addition to users assigned. The roles imply which types of SDDS can be used by users in the projectFutureGrid has some roles but need to exp in addition to This definition supported by FutureGrid [portalUser: Project participantsUsers have individual authorization roles in addition to roles inherited from projects with which they are involvedUsers are assigned to projects by project leadPublic projects can be joined by any Cloudmesh userExperiment: The activity unit as long as CloudmeshSDDS: Software Defined Distributed SystemSDDSL: Specification Language as long as SDDS; essentially exists from various sources Infrastructure: Clusters: Computers, Storage, Network with some reason to be treated as one: Infrastructure has Type as in different Amazon Instance TypesManagement StructureProvisioning rules as long as administratorsUsage rules as long as users of particular rolesA current stateA time interval ranging from transient to a longer term persistence in addition to including a scheduled start timeNote storage could often need to be persistentVirtual Infrastructure: Dynamically defined Slices of InfrastructureFederated Virtual Infrastructure is a Software Defined Distributed System SDDS assigned to a Cloudmesh user as long as an Experiment in a ProjectCloudmesh Definitions IISDDS Software Defined Distributed SystemsCloudmesh builds infrastructure as SDDS consisting of one or more virtual clusters or slices with extensive built-in monitoringThese slices are instantiated on infrastructures with various ownersControlled by roles/rules of Project, User, infrastructureOne needs general hypervisor in addition to bare-metal slices to support FG research The experiment management system is intended to integrates ISI Precip, FG Cloudmesh in addition to tools latter invokesEnables reproducibility in experiments.

Cloudmesh Definitions IIICloudmesh Image: The software that is loaded on an Infrastructure to provision it.For nodes, image is loaded on bare metal or a hypervisorImages created as described belowCloudmesh Image Template: An abstract specification of an Image used to define an implementation that is valid across multiple Infrastructures: three stepsTemplates as a set of one or more scripts/XML specificationsGeneric or base images that can be modified on general devops principles.Host specific ImagesFutureGrid has a prototype Image in addition to Template LibraryNote templates are preferred model as template description is what we mean by Software defined SystemsHowever one may only have an image in some cases in addition to also provisioning speed is improved by taking templates in addition to pre-generating images as long as particular infrastructuresCloudmesh Definitions IVCloudmesh Matchmaker CMPlan chooses appropriate Infrastructures that can be used by CMProv to satisfy a user requested SDDS (not implemented)CloudMesh Provisioner CMProv takes a user request in SDDSL in addition to a chosen Infrastructure in addition to provisions the infrastructure in accordance with user roles, Infrastructures current state, management usage in addition to provisioning rules in addition to generates requested virtual infrastructureCMProv uses appropriate Cloudmesh Images in addition to Templates in addition to capabilities of Cloudmesh depend on availability of appropriate images/templates CMExec produces the users’ requested SDDS as a federation of Virtual Infrastructures created by CMProvCMMon sets up monitoring in addition to experiment management infrastructure (incomplete) CloudMesh Administrative View of SDDS aaSCM-BMPaaS (Bare Metal Provisioning aaS) is a systems view in addition to allows Cloudmesh to dynamically generate anything in addition to assign it as permitted by user role in addition to resource policyFutureGrid machines India, Bravo, Delta, Sierra, Foxtrot are like thisNote this only implies user level bare metal access if given user is authorized in addition to this is done on a per machine basisIt does imply dynamic retargeting of nodes to typically safe modes of operation (approved machine images) such as switching back in addition to as long as th between OpenStack, OpenNebula, HPC on Bare metal, Hadoop etc.CM-HPaaS (Hypervisor based Provisioning aaS) allows Cloudmesh to generate “anything” on the hypervisor allowed as long as a particular userPlat as long as m determined by images available to userAmazon, Azure, HPCloud, Google Compute EngineCM-PaaS (Plat as long as m as a Service) makes available an essentially fixed Plat as long as m with configuration differencesXSEDE with MPI HPC nodes could be like this as is Google App Engine in addition to Amazon HPC Cluster. Echo at IU (ScaleMP) is like thisIn such a case a system administrator can statically change base system but the dynamic provisioner cannot

CloudMesh User View of SDDS aaSNote we always consider virtual clusters or slices with nodes that may or may not have hypervisorsBM-IaaS: Bare Metal (root access) Infrastructure as a service with variants e.g. can change firmware or notH-IaaS: Hypervisor based Infrastructure (Machine) as a Service. User provided a collection of hypervisors to build system on.Classic Commercial cloud viewPSaaS Physical or Plat as long as med System as a Service where user provided a configured image on either Bare Metal or a HypervisorUser could request a deployment of Apache Storm in addition to Kafka to control a set of devices (e.g. smartphones)Cloudmesh Infrastructure TypesNucleus Infrastructure:Persistent Cloudmesh Infrastructure with defined provisioning rules in addition to characteristics in addition to managed by CloudMeshFederated Infrastructure:Outside infrastructure that can be used by special arrangement such as commercial clouds or XSEDETypically persistent in addition to often batch scheduledCloudMesh can use within prescribed provisioning rules in addition to users restricted to those with permitted access; interoperable templates allow common images to nucleusContributed InfrastructureOutside contributions to a particular Cloudmesh project managed by Cloudmesh in this projectTypically strong user role restrictions – users must belong to a particular projectCan implement a Planetlab like environment by contributing hardware that can be generally used with bare-metal provisioningArchitectureCloudmesh Management Framework as long as monitoring in addition to operations, user in addition to project management, experiment planning in addition to deployment of services needed by an experimentProvisioning in addition to execution environments to be deployed on resources to (or interfaced with) enable experiment management.Resources.

Gable, Gene Contributing Editor

Building Blocks of CloudmeshUses internally Libcloud in addition to Cobbler Accesses via abstractions external systems/st in addition to ardsOpenPBS, Chef, Openstack (including tools like Heat), AWS EC2, Eucalyptus, AzureXsede user management (Amie) via Futuregrid Implementing Slurm, OCCI, Ansible, Puppet Evaluating Razor, Juju, Xcat (Original Rain used this), Foreman Details on Cloudmesh FunctionalityDue to its integrated services Cloudmesh provides the ability to be an onramp as long as other clouds.It provides in as long as mation services to various system level sensors to give access to sensor in addition to utilization data. Internally, it can be used to optimize the system usage.The provisioning experience from FutureGrid has taught us that we need to providethe creation of new clouds (rain)the repartitioning of resources between services (cloud shifting) in addition to the integration of external cloud resources in case of over provisioning (cloud bursting)As we deal with many IaaS frameworks, we need an abstraction layer on top of the IaaS framework.Experiment management is conducted with workflows controlled in shells, Python/iPython, as well as systems such as OpenStack’s Heat.Accounting is supported through additional services such as user management in addition to charge rate management.Not all features are yet implemented. Figure shows the main functionality that we target at this time to implement.User in addition to Project Management FutureGrid user in addition to project services simplify the application processes needed to obtain user accounts in addition to projects.We have demonstrated in FutureGrid the ability to create accounts in a very short time, including vetting projects in addition to users – allowing fast turn-around times as long as the majority of FutureGrid projects with an initial startup allocation.We also have shown that we can integrate with other services on user management such as XSEDE, we also have access to the technical team that integrated OSG into XSEDE in addition to the XSEDE TAS projectCloudmesh re-uses this infrastructure in addition to also allows users to manage proxy accounts to federate to other IaaS services to provide an easy interface to integrate them.

Experiment Planning – FutureImagine a shopping cart which will allow checking out of predefined repeatable experiment templates.Cost is associated with an experiment makingClearing house of imagesClearing house of complex deployments.Integrated accounting framework allowing a usage cost model The cost model will be based not only on number of core hours used, but also the capabilities of the resource, the time, in addition to special support it takes to set up the experiment. We will exp in addition to upon the metrics framework of FutureGrid that allows measuring of VM in addition to HPC usage in addition to associate this with cost models. Benchmarks will be used to normalize the charge models.Cloudmesh Provisioning in addition to Execution Bare-metal ProvisioningOriginally developed a provisioning framework in FutureGrid based on xCAT in addition to Moab. (Rain)Due to limitations in addition to significant changes between versions we replaced it with a framework that allows the utilization of different bare-metal provisioners.At this time we have provided an interface as long as cobbler in addition to are also targeting an interface to OpenStack Ironic.Virtual Machine ProvisioningAn abstraction layer to allow the integration of virtual machine management APIs based on the native IaaS service protocols. This helps in exposing features that are otherwise not accessible when quasi protocol st in addition to ards such as EC2 are used on non-AWS IaaS frameworks. It also prevents limitaions that exist in current implementations, such as libcloud to use OpenStack.Network Provisioning (Future)Utilize networks offering various levels of control, from st in addition to ard IP connectivity to completely configurable SDNs as novel cloud architectures will almost certainly leverage NaaS in addition to SDN alongside system software in addition to middleware. FutureGrid resources will make use of SDN using OpenFlow whenever possible though the same level of networking control will not be available in every location. Provisioning – Cont’d Storage Provisioning (Future)Bare-metal provisioning allows storage provisioning in addition to making it available to usersPlat as long as m, IaaS, in addition to Federated Provisioning (Current & Future)Integration of Cloudmesh shell scripting, in addition to the utilization of DevOps frameworks such as Chef or Puppet.Resource Shifting (Current & Future)We demonstrated via Rain the shift of resources allocations between services such as HPC in addition to OpenStack or Eucalyptus. Developing intuitive user interfaces as part of Cloudmesh that assist administrators in addition to users through role in addition to project based authentication to move resources from one service to another.

Cloudmesh Metric ComponentsData Collector in addition to Metric service. One of the fundamental services needed is a data collector. It collects relevant data from a variety of sources including resource databases, log files, in addition to data reporters. Hence, to integrate new cloud services into the data collectors we have to define a data model, as well as data sources that populate the data model with data. Currently, data collectors are available as long as OpenStack, Eucalyptus, in addition to Nimbus but not limited to these plat as long as ms. Dependent on the IaaS framework they obtain the data from different sources such as log files or databases. Beneficial in as long as mation to be collected includes detailed in as long as mation about the virtual machines (VM), the users in addition to /or projects starting them, memory usage over the lifetime of the VMs, errors associated with VMs during runtime, or at startup. One of the issues to be addressed is, if such data should be directly accessed in the production environment offered by the IaaS framework. Practical experience with FutureGrid has shown that the analysis of the data poses a significant amount of stress on the originating resource, making it impractical to offer a detailed report in addition to metric system on the original data sources. Hence, it is important that we replicate it when the in as long as mation we request is involved in a detailed analysis. For some smaller scale queries, as the one posed by most users, direct access is sufficient in addition to desirable as long as the live view of the system in order to provide in as long as mation about how many VMs are currently running, on which system in addition to by whom.Cloudmesh Metric ComponentsMetric Analyzer. The data collected provides the opportunity to analyze it as long as specific needs in a repeated fashion or provide filters in addition to services as long as further specialized analysis. The FutureGrid metric framework there as long as e provides a metric analyzer component with a convenient interface as long as analyzing the data not only on an automated fashion, but also interactively through a simple metrics analyzer shell. In as long as mation of interest includes yearly, monthly, in addition to /or weekly usage in as long as mation by user, project, resource, provider, in addition to the agglomerated in- as long as mation. Our scripting environment provides this in as long as mation in addition to is run at predefined intervals or upon request. In the future, we will be enhancing the service to allow users to schedule queries to conduct specific analysis. To avoid repeated analysis, metric result caching is conducted. Thus, if a query has been executed in the past the result is cached in addition to returned without reanalysis (if not as long as ced). To more easily facilitate fast in addition to distributed calculation of the results by multiple users, we will base future versions of the Cloudmetric system on NoSQL database technologies.Accessing Cloudmesh MetricsEarly on we recognized that the access of in as long as mation in addition to the metrics must be provided through a variety of interfaces. This includes comm in addition to shells, programming API’s, REST interfaces, graphical user interfaces, in addition to printable reports.Interactive Comm in addition to Shell. To simplify the interactive use, we have developed a python comm in addition to shell called CMD3 that allows the dynamic load of additional comm in addition to s, thus making it ideal to define new analytic methodsREST API. We are currently building access through a convenient REST API to allow easy access from Web frameworks, but also integration from arbitrary programming languages Programming API. We have provided a robust API interface in python to access the basic analytical functions useful as long as many users in addition to reused by the interactive comm in addition to shell in addition to the REST service.Graphical Representation in addition to Printable Reports. Using our basic API in addition to comm in addition to shell, we have integrated them into the Python sphinx framework to expose the metric data in a convenient as long as m in addition to present the data online via charts or in a PDF report. As sphinx offers the export of data reports in PDF, we leverage this framework in addition to do not have to develop a separate framework as long as it. The sphinx framework service is currently enhanced, allowing customizable interactive queries to the metric in addition to data sources. The data can be represented visually in various chart as long as ms such as bar graphs, line charts, or pie charts. A template as long as generating a quarterly in addition to yearly report of the data exists making adaptation to additional resources or other provider consortia easy. Furthermore, the data can be exported in a variety of as long as mats such as JSON or CSV making it possible to use other tools, such as excel as long as data post processing.

Gable, Gene Contributing Editor

Gable, Gene is from United States and they belong to and they are from  Petaluma, United States got related to this Particular Journal. and Gable, Gene deal with the subjects like Communications; Digital Graphics and Imaging; Graphic and Commercial Design; Website Design

Journal Ratings by Albright College

This Particular Journal got reviewed and rated by Albright College and short form of this particular Institution is PA and gave this Journal an Excellent Rating.