Cloud Based Compute Solutions
One of the most popular reasons for businesses moving to the cloud is for the use of compute power. This can be for migrating existing on premise workloads or transitioning to a cloud based virtualisation platform. Another reason with be the use of high performance computing. We will discuss some of the various compute options in this blog post.
Compute
Running compute power in the cloud is becoming the standard way to run virtual servers, application services as well as high performance computing. Using older approaches such as onsite data centres and servers with virtualisation has become costly and underused. Often the capacity is not fully used and there is also the cost of hardware maintenance.
By migrating your existing compute workloads to the cloud you can reduce cost, maintenance and increase your reliability and up time. You can migrate your virtual machines (VM's) directly into cloud platforms with minimum or no downtime. Once in the cloud you can then only focus on the management of the software layer and not hardware.
These days its become fairly standard to migrate existing on premises VM's into the cloud despite which hypervisor platform you are using i.e VMware or Hyper-V. You are also able to Physical 2 Virtual physial servers and migrate these to cloud. Please be careful to check your chosen cloud platforms operating system requirements prior to starting any migrations.
HPC High Performance Computing
Another very interesting use of cloud computing is for high performance computing (HPC) using large scale batch compute tasks to run huge loads like rendering. This can be much easier and cheaper than trying to build and run this in an on premise data centre. In the cloud you only pay for what you use and not the spare unused capacity. In HPC you may not be constantly running these workloads so you will not pay for unused compute time.
IaaS and PaaS
You are able to run compute functions in Infrastructure as a Service (IaaS) whereby you manage and look after your own infrasructue or VM's or via Platform as a Service (PaaS) were you can directly spin up specific functions like web apps that run in the cloud. IaaS may be more suited to a business looking to import existing infrastructure into the cloud and PaaS may be better suited to developers not interested in maintaining any infrastructure.
Containers
Containers are pretty much the future of running applications in the cloud (or even on premises) and with using solutions like Docker and Kubernetes the process of deploying containers has become much easier. In the cloud you can use the relevant cloud platforms Kubernetes engines to spin up containers in seconds and to mange them going forward. Google Cloud's GKE is really leading in this space with Microsoft's AKS which has just become GA (Generally Available) quickly catching up. I will be writing a blog post post about Containers in the near future as this is becoming a really hot topic in cloud computing.
Serverless Functions
Serverless and functions are used by developers to write and run code immediately and not have to deal with any servers or infrastructure at all. Code can be directly run and actioned without any need to worry about capacity planning or server management. Of course there are servers required but these are all managed by the cloud platforrms.
Serverless Functions
Serverless and functions are used by developers to write and run code immediately and not have to deal with any servers or infrastructure at all. Code can be directly run and actioned without any need to worry about capacity planning or server management. Of course there are servers required but these are all managed by the cloud platforrms.
https://cloud.google.com/products/compute/
https://azure.microsoft.com/en-us/product-categories/compute/
Cloud Based Virtual Networks
While discussing the implementation of cloud infrastructure we touched on the base of core infrastructure that will need to be provisioned in the cloud in order to begin your migration or greenfield cloud deployment. The core infrastructure required is cloud storage, compute resource and virtual networks. Today we will be going into virtual networks in more depth.
Virtual Networks
As previously discussed virtual networks are absolutely vital to understand in order to correctly provision and deploy cloud resources. These virtual networks are used to connect, segment and link your cloud based resources as well as on premises networks.
Virtual network segmentation is completed with the use of subnets. These subnets are used to seperate various cloud based networks using different IP ranges. These are seperate networks that can be used for different functions, for example the front end and back end tier of an application.
Another aspect to understand is the use of firewalls within these virtual networks. Traditionally a firewall is used on premises between the internal network and external network (internet). In cloud services a firewall is used in the same way but also internally, so you are able to secure internal as well as external networks and applications.
As mentioned above another incredibly important aspect of cloud based virtual networks is the ability to connect to your on premises environment. This is valuable in order to build and architect hybrid cloud solutions. This can be achieved by using direct ISP based links that bypass the public internet to keep things internally secure and fast or via VPN tunnels.
Cloud Platforms
There are some differences between the different cloud providers but basically cloud based virtual networks are quite similar and perform reasonably the same function. In Azure you get Virtual Networks which are the base of your cloud based network infrastructure and in Google Cloud Platform you get Virtual Private Cloud Networks (VPC's) both offer a few different features so its always important to read the relevant documentation and use a trial account to test for your specific requirements.
To sum up though you need to brush up on your general network skills (get out the Network+ manual) before planning your cloud based networks and I highly recommend some courses in the cloud platform you are looking into going with. There are literally loads of in depth free video courses available on any platform to learn the necessary skills.
Training
Please see below Coursera GCP and Pluralsight Azure on demand courses:
https://www.coursera.org/specializations/gcp-architecture
https://www.pluralsight.com/courses/planning-designing-microsoft-azure-network-solutions?twoid=e7d045ab-0691-4def-896a-8db6cb74790b&aid=7010a000001xDURAA2
Implementing Cloud Infrastructure
To break it down this is very comparable to on premises infrastructure as when looking to physical storage, physical servers, virtualisation, virtual networks and virtual machines. The cloud reduces the need have the physical infrastructure in place. You are able to utilise this on a pay per use model in any of the public cloud providers for example Google Cloud Platform or Microsoft Azure. You are charged for what you use which is great!
Virtual Networks
After activating your cloud subscription you can begin setting up your virtual network. This has different names depending on what provider you are using i.e in Microsoft Azure its called Virtual Networks and in Google Cloud Platform its called Virtual Private Cloud Networks (VPC's) Basically these are similar ways to perform network segmentation in the cloud based on virtualised networks. Subnets are used to segment these virtual networks or VPC's. You are also able to integrate load balancers and firewalls:
Within these virtual networks you can isolate specific services, i.e virtual machines, you can implement load balancers as well as connect networks from different regions togeather. You are also able to implement security with firewall's across these virtual networks both internally and externally. Another feature of virtual networks is the ability to connect them with your existing on premises networks. There are various methods available in order to achieve this as in a direct Interconnect (GCP) or Express Route (Azure) link from your site to the applicable cloud provider. Another way do to this is be using secure encrypted VPN tunnels:
Cloud Storage
Cloud Storage is absolutely critical as this is where all of your resources will be stored in the cloud. I have previously written about cloud storage if you would like to read further:
In the context of this article we will be referencing more towards storage of infrastructure like virtual machine files, virtual machine disks and general file storage. These will be the locations where your compute workloads will be stored when created. There are various different types of storage but for virtual machines you will look at options like HDD or SSD depending on workload. Google GCP has persistent disks and Azure has managed disks for VM's. http://www.ruckcloud.ml/2018/04/lets-talk-about-managed-disks.html
Compute
The compute layer is all about the computing resources that you will be utilising. This is based on virtual machines in one form or the other. You can spin up traditional VM's one at a time with a large selection of different operating systems from Windows to Linux. These are called IaaS (Infrastructure as a Service) VM's. You can also leverage batch operation with automating a large number VM creation to achieve a large processing job for example. These VM's can automatically scale up and down based on load and you are only charged when they are in use. With IaaS you have direct control and management of your VM's.
You are also able to make use of virtual machines in PaaS (Platform as a Service) where you can immediately spin up app's for computing needs without the need of managing IaaS VM's. This is very handy for developers that are not too concerned with managing VM's.
In this article I have touched on the core base infrastructure required with cloud computing. These areas all go into much more depth but sometimes its nice to get a simple overview of what they are and how they work. This is really essential to understand when first looking into the cloud to either build new services or migrate your existing infrastructure.
Further information on cloud infrastructure is available at:
Lets talk about managed disks
Managed disks are Azure managed virtual machine disks that are easily added during virtual machine (VM) builds. When activating the managed disk its added to your VM in replacement of the traditional storage account based unmanaged disk. Originally this was the only way of doing this as all VM disks had to be placed into storage accounts. Adding a managed disk though VM creation is a very easy process and literally takes one click:
There are many advantages for using manged disks opposed to traditional unmanaged disk storage which are mainly related to less overhead management, less resource sprawl, secured disk storage, better high availability and reliability on virtual machine disk storage.
Simplified Management - You can specify the type of disk and size needed and Azure will automatically create and manage the disks for you.
Scalable virtual machine deployments - Create thousands of managed disks within minutes. Create up to 1000 virtual machines in scale sets in a single large cluster.
More Secure - Using Azure RBAC (Role Based Access Control) you are able to create granular role based access control on your managed disks.
Highly durable and available - Your data is replicated simultaneously to three different replicas. If one replica fails there are two others to take over.
The below is a great comparison between unmanaged and managed disks:
There are also various further advantages of using managed disks. Namely multiple storage options like SSD premium managed disks for critical performance intensive workloads, and HDD for standard managed disk non-critical workloads. Easy migration from standard to premium managed disks as well as your existing ARM (Azure Resource Manager) virtual machines into managed disks. Point in time backup snapshot of your managed disk to create new managed disks later. Simple custom image management and encryption with bringing your own keys is available.
Managed disks are now the best way to use virtual machine disks in Azure. It's actually much easier to roll out and less admin than unmanaged disks as well as more reliable.
Managed disks are now the best way to use virtual machine disks in Azure. It's actually much easier to roll out and less admin than unmanaged disks as well as more reliable.
Please read the following for additional information:
Is Cloud Certification worth it?
Cloud skills are absolutely essential for anyone working in IT, from developers to operations engineers you need to rapidly skill up in order to handle the changes in the industry. Most on premises workloads are currently being migrated or plans are being made to migrate shortly.
Where Cloud Certification itself fits in is being able to provide the necessary training and exams to validate these Cloud skills. By getting cloud certified this can assist you with gaining professional credibility as well as improve your hiring and promotion chances.
In my opinion the best Cloud certification paths are with the two biggest currently growing cloud providers in the market. Microsoft Azure and Google Cloud Platform. These providers are in the process of quick growth and constantly growing their product portfolios.
Lets discuss the certification options provided by both Microsoft and Google. Both vary slightly in exams and certification paths. Microsoft's are more complicated requiring a series of exams to achieve various certifications... MCSA Cloud Platform as well as the MCSE Cloud platform and Infrastructure. I will be focusing on these certifications as coming from an infrastructure background in an operations team or technical architect experience. There are other cloud certifications related to data science, machine learning and big data.
Microsoft Azure
Microsoft's Cloud certification for Azure starts with the 70-533 exam. This is related to implementing Azure Infrastructure Solutions. Once this is achieved you will require one more exam from the 70-532, 70-535, 70-473, 70-475 in order to achieve the MCSA Cloud Platform. However this is the path that I took but any two of these exams will count towards the MCSA Cloud Platform certification. If you would like to go further to achieve the MCSE Cloud Platform and Infrastructure you will need to achieve an additional elective exam also from this list. So to achieve the MSCE in Cloud Platform you will need to pass 3 exams.
Microsoft has partnered with Acclaim which provides badges for your passed exams as well as certifications. Microsoft also provides downloadable certificates from your MCP profile.
The Acclaim badges can be used to verify your certifications:
Google Cloud Platform
Google makes certification a lot more simple and only one exam is required in order to become Google Cloud Certified Professional Cloud Architect. However this exam is high level and contains a very large course content. Once passed you will become certified. This is not an easy exam and is comparable to Microsoft's 70-535 Architecting Azure Solutions. These are both architect level exams that require strong cloud architect knowledge.
Google provides certificates for your achievement that are verified by credential.net
I strongly recommend certifying across different cloud providers. The reason for this is that a lot of businesses these days are adopting multi-cloud deployment strategies which means you are quite likely to find multiple cloud providers in production or even cross platform migrations. Being able to work across the various cloud providers is becoming essential.
Please find below additional reading and information:
What exactly is Cloud Storage?
A good way to explain cloud storage is by comparing consumer vs commercial cloud storage. Most people these days are actively using and relying on cloud storage everyday through One Drive, Google Drive, Dropbox and the large number of other providers. Basically all these options are the same, store your media, images, files offsite in the cloud.
The commercial aspects of cloud storage are a bit more complex as you will need to identify your workload and requirements of it. As in blob storage is great for videos, images and media but that will not work with databases or application back end data. The various cloud suppliers have various tiers of cloud storage to help you identify your requirements. These can range from "hot" storage or frequently accessed data to "cold storage" archive data.
Looking at the Microsoft Azure offerings the below different products are available:
File
Simple, distributed, cross-platform file system
- Lift and shift migration
- Simple and inexpensive
- Move data to cloud with no coding
Disk
Premium storage for I/O-intensive applications
- Low latency, high throughput
- Automatic triple replication
- Enterprise-grade durability
- Learn more
Blob
Massively-scalable object storage for unstructured data
- Cost-effective for massive volume
- Tiered storage options
- Single infrastructure with global reach
- Learn more
Queue
Durable queues for large-volume cloud services
- Simple, cost-effective messaging
- Decoupled component flexibility
- Resilient scaling and buffering
- Learn more
Table
- Flexible NoSQL database
Key-value table storage
- Structured or unstructured data
- Low latency at Internet scale
- Learn more
Archive
Low cost storage for rarely used data
- Data automatically encrypted at rest
- Seamless integration with hot and cool storage tiers
- Supported by leading Data Management partners
Most cloud providers fit into the same above categories with different service offerings and names, Google Cloud Platform for instance offers the following in Google Cloud Storage:
Please see bottom of the article for links with further details and information.
A good way to start with cloud storage options are with your backup or archival data. By moving this data to the cloud a company can start leveraging the power of cloud without initially moving their entire workload. By storing your backups and even running your daily, weekly or monthly backup runs into the cloud this will provide a cost effective and reliable offsite backup option that can really prove to be invaluable in the case of a disaster.
There are really a huge volume of different cloud backup providers these days which all basically offer the same thing with different types of software. Locally running backups that copy your data to the cloud. All of these products will use either their own cloud storage servers or more likely one of the big cloud providers. You might even be able to get the choice of which public cloud storage provider you use with the specific backup product. You could also go directly to a public cloud provider and use their native cloud backup tools.
There are also on premises to cloud replication devices such as Microsoft's Storesimple that will automatically move your archival or "cold' on premises data into the cloud. This is something that can assist with large enterprise storage volumes and cloud storage adoption.
Ultimately cloud storage is expanding rapidly with the great volumes of data being generated everyday and it is really the future of all data storage. Going forward it will be easier, safer and more cost effective to utilise and migrate your current on-premises storage into the cloud. A good way to start on this journey is with your backups.
Further sources and information:
Virtual Machine Scale Sets
- Create thousands of identical virtual machines in minutes
- Quickly scale your big compute and big data applications
- Rely on integrated load balancing and autoscaling
- Attach additional data disks as per your application requirement
- Deploy virtual machines and updates at scale
- Support Linux or Windows images and extensions
- Run Cassandra, Cloudera, Hadoop, MongoDB, and Mesos
- Deploy across availability zones to protect against datacenter failures
Virtual Machine Scale Sets allow you to deploy and manage sets of identical auto scaling virtual machines. You are able to auto scale the virtual machines manually or by using certain metrics as in CPU usage,memory usage or network traffic.
When creating virtual machine scale sets via the Azure Portal a load balancer is created in order to spread inbound requests across the available virtual machines. This in turn allows you to distribute your application across multiple instances for high redundancy.
On creation you are able to add operating system disk images and virtual machine sizes:
Virtual machines scale sets provide the following features:
- Easy to create and manage multiple VM's
- Provides high availability and application resiliency
- Allows your application to automatically scale as resource demand changes
- Works at large scale
Further detailed information on Microsoft Azure documentation:
Detailed technical guide on creating virtual machine scale sets via the Azure Portal:
Virtual Machine Scale Sets:
Architecting Azure Solutions 70-535
The 70-535 replaces the older 70-534 exam and is an elective to the MCSE Cloud Platform and Infrastructure certification track. Along with another one of the 532, 533, 473 or 475 exams this will also earn you the MCSA Cloud Platform certification.
First things first if you have not had any recent Microsoft Cloud exam exposure you are in for quite a difficult journey as these new certifications are very tough and require a lot of dedication. These exams are difficult and require extensive in-depth Azure experience.
With regards to the 70-535, the course content was revised after the 70-534 exam was retired at the end of 2017. Anyone studying on that path needs to check the new official exam guide to verify the latest material. There have been a lot of additions as per the latest product implementations in Azure. I will link the exam guide article at the bottom of this post.
The following course content is currently covered on the 70-535:
Each of these sections are very in depth and go into a lot of detail. Please note that you will require extensive experience in all of these areas before attempting the exam. The best way to do this is by gaining Access to Azure and labbing out all of the work. Practical experience, deep understanding and real world experience is vital for this exam. Azure does offer a free account with $200 credit as well as student access options to get this practical knowledge.
To sum up all resources that I used while training and studying for this exam, I have put together some links below to all free training documentation, free training videos as well as useful links that can be used. Honestly using all of the resources below, getting free Azure access and reading the relevant documentation will assist hugely with this exam.
This link is a great place to start and a great resource that puts all of the course content into a well structured flow of resources based on course content and I highly recommend this:
Please find further invaluable links to free training resources and videos:
This is really a challenging exam and please take your time in preparation. There are also some articles on this blog with some resources related to this exam so they would be worth a read even if it's just an overview of some of the content. And yes I did thankfully pass! :)
Globally distributed Azure #CosmosDB
The application solutions best suited for Cosmos DB are: web, mobile, gaming and IoT that needs to handle a massive amount of data, reads and writes at a global scale with near-real response times. These applications will benefit from guaranteed high availability, high throughput, low latency and tunable consistency. Cosmos DB is a NoSQL database.
Azure Cosmos DB has various API's for accessing and querying data: SQL API, Mongodb API, Cassandra API, Graph (Gremlin) API, Table API.The data model Cosmos DB is built on is: atom-record sequence (ARS) which supports multiple data models including document graph, key-value table and column-family data models.
A great preview feature of Cosmos DB is that you can try it for free for up to 7 days without having an Azure subscription or any credit card details which is a nice touch. This is great for creating and testing between the following API and data models: SQL, MongoDB, Table and Graph. https://azure.microsoft.com/en-gb/try/cosmosdb/
If you already have an Azure subscription, Azure Cosmos DB can be launched easily from the Azure portal and to create a DB is literally a one click solution, there is no infrastructure configurations required at all, you only need to select which API you require & create:
Further detailed information and resources:
Azure Application Gateway Introduction
The Azure Application Gateway also provides the following features: http load balancing, cookie based session affinity, SSL offload, end to end SSL, URL based content routing, multi-site routing, websocket support, health monitoring, SSL policy and ciphers, request redirect, multi-tenant back-end support and advanced diagnostics.
As mentioned above, the Azure Application Gateway is used for layer 7 load balancing. Using Traffic Manager to distribute the traffic to various application gateway services which in turn provide layer 7 load balancing. There are three different types of load balancers in Azure: Azure Load Balancer - Layer 4 traffic distribution, Application Gateway - Layer 7 load balancing and Traffic Manger which uses DNS to direct traffic distribution to endpoints based on an end user location. Traffic Manager can be used in conjunction with Application Gateway:
Further detailed information can be viewed at: https://docs.microsoft.com/en-gb/azure/application-gateway/application-gateway-introduction
Azure Security Center Playbooks Introduction
Azure Security Centre is a great tool in order to manage your hybrid cloud security health. As I have previously written its a tool that can be used to monitor both Azure infrastructure as well as on-premises machines with a central monitoring dashboard. Please feel read to read the other blog post in the Azure Security Centre series. http://www.ruckcloud.ml/2018/02/using-azure-security-center-for.html
Today I will be writing about a specific feature of the Azure Security Centre called Security Playbooks. Security Playbooks are alert triggered procedures that run within Azure Security Center. When a specific alert is is triggered a playbook will run that will automate a response to that specific alert. This can help orchestrate as well as speed up security alert management.
Security Playbooks are based on Azure Logic Apps. What this gives us is the ability to use security templates based in Azure Logic Apps which you can edit or create and then trigger these with Security Centre.
What are Azure Logic apps?
Azure Logic Apps help to simplify and implement scale-able integrations and workflows in the cloud. Please see further info at the end of this post.
Creating a Security Playbook
Please see the below steps in order to create a Security Playbook in the Azure portal:
1. Select Security Centre - Playbooks (preview) - Add Playbook
4. Click on the newly created logic app which will launch the logic app designer:
5. Click blank logic app:
7. Add an action to run and click save:
8. After this has been completed, the playbook can be run from Security Center "Playbooks"
Please read further on Azure Logic Apps: https://docs.microsoft.com/en-gb/rest/api/logic/
Using Azure Security Center for monitoring
You are able to apply policies and also locate and fix vulnerabilities before they can be exploited. What is also great is that you are able to leverage advanced analytics to detect and mitigate attacks on your infrastructure using the Advanced Threat Protection module.
In this article I will go about explaining the basics of this service, from locating where to find it to some basic overviews of the monitoring dashboard and some of the remediation features. As with all my articles they can be read by the first time Azure user all the way up to a seasoned pro who may be interested in learning about a new product.
The Azure Security Centre is simply launched from within the Azure portal:
Once you have opened the portal, it will display your monitoring dashboard with alerts:
This dashboard will contain your security related overview, prevention and detection. Please note that you will require an additional 60 day trial to view the Advanced Threat Detection. the Advanced Threat Detection is an advanced feature that can be used to automatically locate and resolve security issues based on the Azure Security Centre intelligence.
Recommendations
When monitoring the dashboard you can scroll through and check the recommendations that can be implemented to better secure your current workload, these are arranged into a high and medium severity. When opening these they will provide instructions on how to resolve and implement better security practises onto your cloud or on-premise workloads.
As per the the above recommendation I have located that I am not using a Network Security Group (NSG) on a specific virtual network. I can then click through the blades in order to find more information on this specific issue and resolve it directly from within the portal. As per the below I can directly enable the required NSG through the next portal steps:
Prevention
Under the prevention tab you will see additional alerts generated based on your infrastructure. These tabs will contain compute, networking, storage & data and applications.
In my example I have a few alerts across compute and networking. When clicking on the compute tab I am given the following recommendations with regards to my virtual machines:
From within this tab I can continue through to obtain further information as in this case I have not installed the endpoint protection on the virtual machine (this is used to scan for malware threats) as well as having missing disk encryption on the virtual machine disks. Within the following windows you will receive further information and directions for resolving these issues. This allows you to become proactive with regards to security issues.
Within networking, storage and data similar issues are reported with further information on the threats and directions on how to resolve them. This is incredibly useful and provides a great overview of your Azure infrastructure security and gives great direction on resolving any issues that may have been located.
One other point that I haven't mentioned is that you can also install the Azure Security Centre monitoring agent on on-premises machines which create a fully hybrid cloud security monitoring overview. Its also great as you can have all security information for cloud and on-premises in one place (dashboard) for monitoring and reporting. A Log Analytics work space is required to on-board non-Azure computers to Security Centre:
There are a lot more functions that can be enabled within Azure Security Centre and I have only touched on the basics in this article. Please feel free to look further into Advanced Threat Detection which contains great threat intelligence and security alerting functions.
Further information can be found here: https://azure.microsoft.com/en-gb/services/security-center/ and documentation here: https://azure.microsoft.com/en-us/documentation/services/security-center/
Subscribe to:
Comments (Atom)
