Azure app service is a quasi PaaS and IaaS type of solution. Most importantly, it can remove the idea of a DMZ and put the power of public internet access into the hands of developer, remove separation of duties and most interestingly create a back channel for malware command and control systems.
Plus there are a few bugs in the current system that general users should be aware of. If you don’t know about it, it’s worth learning and digging into at your current company.
This post skips around bit doing a couple open source “Hello World” examples and then digs into the Azure documentation ….. this is a collection of learning from some personal research I was doing to prepare for some upcoming projects at work. Everything here uses open source code and public Azure tutorials and examples…
- A few simple tutorials on using Azure secure app-service infrastructure and some security consequences
- The tutorial uses Microsoft provided Java . Node.js “Hello World” example to demonstrate important key design considerations and findings
- The example also uses some manual deployment Web UI examples
- ARM JSON code to deploy a similar set up in dev can be found in SRC folder
- Azure ARM deployment JSON source can be found in the SRC folder for illustration
- The SpringBoot Hello World application is found in main-zip
Before we begin
- Any of your own personal demo’s should be performed in a development environment isolated from production systems and production data.
INSECURE Pricing Plan / Hosting – Configuration (CLI Java example)
git clone https://github.com/spring-guides/gs-spring-boot
cd gs-spring-boot/complete
mvn com.microsoft.azure:azure-webapp-maven-plugin:2.1.0:config
Use the java code to do the walk through and stop here ….
During the CLI deployment process that you’ll be asked about the PricingTier. The default is Pl2 but numerous other options are listed.
This can be confusing for new developers who just see an easy and cheap way to deploy their code. However, the pricing tier goes beyond simply choosing a cheaper or more expensive option.
This decision affects both your scalability, network performance and privacy of your application code, and how accessible it is from attack on the internet and more ….
As of 10/25/2021, the Java SDK doesn’t even offer the option for the “ISOLATED” option, meaning you may accidentally choose a less secure option without noticing.
The above picture demonstrates the added complexity and pros/cons of the pricing tier decision. What appears to be a very simple development decision actually has many other consequences.
- Shared compute: Free and Shared, the two base tiers, runs an app on the same Azure VM as other App Service apps, including apps of other customers. These tiers allocate CPU quotas to each app that runs on the shared resources, and the resources cannot scale out.
- Dedicated compute: The Basic, Standard, Premium, PremiumV2, and PremiumV3 tiers run apps on dedicated Azure VMs. Only apps in the same App Service plan share the same compute resources. The higher the tier, the more VM instances are available to you for scale-out.
- Isolated: This tier runs dedicated Azure VMs on dedicated Azure Virtual Networks. It provides network isolation on top of compute isolation to your apps. It provides the maximum scale-out capabilities.
From the Web UI
Navigate to the Web UI –> App Service –> Create –> Click Pricing Plan at the bottom.
You can also visualize the hosting options there if the SDK or CLI does not have the option.
WHY DOES IT MATTER?
- All pricing tiers except Isolated expose the java “Hello World” to the general internet. This means during development it is more likely to expose code to more attacker (s). To be fair, it is possible to isolate client traffic using public internet IPs but not to private networks IPs
- This development decision removes separation of duties of critical internet hosting decisions
- This “Hello World” application may have needed a Web Application Firewall, Code Analysis or on-premise network connectivity planning before launching directly to internet, but the governance the ensure that happens is removed.
- When internet pricing tier (s) + hybrid connectivity is used, you may be creating a backdoor that allows attackers to access other production systems and data
DESIGN ADVICE
- GOOD: Chose Isolated hosting / pricing tier option and host your app-service on private VNET over a a corporate VPN or ExpressRoute etc. etc.
- NOT AS GOOD: Choose the production hosting option and isolate to your specific public workstation client IP / corporate proxy IP and do not use Hydrid Connectivity back to production VNETs or on-premise. this may be okay for public web apps but it may still create a backdoor into on-premise and give one role excessive responsibility.
- BAD: Choose Production or Dev/Test pricing tiers and leave your development code exposed to the internet. Combine the Production or Dev/Test pricing tiers with Hydrid Connectivity back to production VNETs or on-premise
Hybrid Connectivity – backdoor for malwar and data exfiltration
A potentially higher risk development option that isn’t covered in the “Hello World” tutorial is “Hybrid Connectivity”. You can learn more about hybrid-connectivity below
- https://docs.microsoft.com/en-us/azure/app-service/app-service-hybrid-connections
- https://docs.microsoft.com/en-us/azure/azure-relay/relay-what-is-it
On the surface it looks simple enough, just a few CLI commands and some powershell or .Net code …. and you have a backdoor onto Azure without isolated network access or network monitoring from your on-premise network …
az appservice hybrid-connection set-key --hybrid-connection
--key-type
--namespace
--plan
--resource-group
[--subscription]
Think of this as a native “reverse shell” feature. This back-channel could be used by employees to create reverse shells from on-premise to the cloud to access AD or it could equally be used by malware.
Unfortunately, because everything is to Azure Cloud over TLS it’s going to be very hard to detect network based IoCs for this back channel. Knowing “which azure account” and “whats being sent” has to be caught at the process and file system level and correlated with azure cli hybrid connectivity commands.
WHY DOES IT MATTER?
- This development decision can create a backdoor for internet exposed code/logic to communicate with production or on-premise networks and systems
- If the code is vulnerable, exploited then the Hybrid-Connection could be abused to remotely execute code
- This development decision removes separation of duties of critical internet hosting decisions
- This feature creates a back channel for malware command and control and that is going to be difficult to detect
DESIGN ADVICE
- Chose Isolated hosting / pricing tier option and host your app-service on private VNET over a corporate VPN or ExpressRoute. Choose the internal ASE ISB option, more info in URL below
https://docs.microsoft.com/en-us/azure/app-service/environment/network-info
CICD with App Service
So far I’ve only gone over some infrastructure elated points…. When you need to deploy your code into the app service product it offers you a few options. There are various options from
- Native CICD webook where SCM pushes code into app service
- Sync option where App Service reaches out to the SCM and reads
- Local CLI
- FTP
- Creating a locally hosted Git on App service to pushing your package from local machine
Important – FTP – local accounts
There is an option for the use of FTP to manage the release of code to the app service PaaS. This configuration is exposed to the internet when using any other pricing tier other than ISOLATED. Not only could the FTP port be exposed too broadly but it can also be configured with basic AuthN that can be changed by the end user via CLI which may create easilly guessable backdoors.
There are two options, to user user local accounts or use application specific scoped accounts for FTP. Application FTP accounts options are less risky because their permisions are are more granularly scoped.
BUG ALERT
Passwords in ClearText
After you integrate with the external Git using the isolated pricing tier. The password will be displayed in cleartext on the UI to anyone with access to your Azure app service. And when you go to fetch new code as part of the CICD process, your passwords will show up in the logs too.
CICD Design Advice
- BETTER: Use privately hosted CICD agents/runner on secure VMS to call the app service API on isolated pricing tier hosted in internal VNET. Disable FTP
- GOOD: Use application scoped credentials via Git Web Hook to to call the app service API of an isolated pricing tier PaaS hosted in internal VNET. Use FTP application scoped credentials on isolated pricing tier PaaS hosted in internal VNET. Meaning “Call out to the App service API from a more trusted client” … not the other way around …. that’s how passwords are stored in cleartext
- NOT AS GOOD: External Git integration using your corporate user account creds in the username password form fields to perform a git fetch into the app service platform…. If you do this, your corporate creds will be displayed in Azure for all to see.
- BAD: Choose Production or Dev/Test pricing tiers and leave your app-service FTP port exposed to internet and use weak user account local passwords
Kudo – HIDDEN Management Plane
The App service comes with an additional management control plane outside of the core “Azure” web portal. This appears to be more similar to an admin Web UI and Web SSH terminal of the app service virtual appliance… meaning there is a seperate URL that allows you to perform SSH and other functions.
The kudo app is only single factor protected with a local account and also is the core API which exposes the management plain within the VNET … it does appear this platform requires the Azure Okta log-in flow for Web UI … however this may be bypassed… I haven’t tested that just yet ….
Threat Detection
Azure app service is a managed PaaS platform. This is important to your application security because traditionally there would have been server based security controls as part of your application stack. Technologies that detected malware, viruses and forward logs and security events externally to be monitored. Azure app-service does not come with the ability to detect threats by default.
- If you choose to use azure app service you may be using a product, potentially hosted on the internet without any way of detecting whether your code is being actively attacked or exploited
- If your using hybrid connections back to on-premise or to production systems, you may not only have a backdoor which makes the lack of detection a even bigger concern
Design Advice
- Confirm whether your Azure account has any type of detection and response capability. For example, the security team may be using Azure Defender, using a Web Application Firewall or have custom logging and monitoring built from “Application Insights”
- If you’re app-service is not protected with any form of threat detection then using the isolated options makes even more sense
summary
- Consider using the Isolated hosting option with private VPN/Express route access
- Avoid using the Hybrid connection feature, it may create a less secure communication path that avoid other governance processes and security controls
- Check with your IR team on whether you are prepared to detect command and control over these Azure “relay” and “hybrid connection” channels
- Don’t use the CICD feature to “pull” or “fetch” data via App-service. Instead, initiate the connection to the API via a a secure Jenkins worker node or Gitlab runner instance. Otherwise, your passwords may end up in the solution.
- Disable the FTP CICD endpoint if your not using it and disable it when hosted on the other pricing tier models to avoid a back channel into your application stack
- Check whether Azure Defender is enabled on your Azure app service at a minimum and whether anyone is monitoring you solution for active threats and attacks
@secSandman 