Jekyll2022-10-18T09:48:39+00:00https://danielmeixner.github.io//feed.xmlDevelopersDevelopersDevelopersDevelopers.netThis is the place where Daniel Meixner is blogging.A layered approach to sustainable application design2022-09-30T00:00:00+00:002022-09-30T00:00:00+00:00https://danielmeixner.github.io//A-layered-approach-to-sustainable-application-design<h1 id="tldr">tl;dr;</h1>
<p>There are multiple areas where we can adjust applications to reduce carbon emissions. Some of them might not be relevant to you, others might be out of your authority. It’s good to have an understanding that those areas exist - this is where I think the layered approach can help.
At this point (October 2022) there’s no wholistic list of recommended patterns to build more sustainable applications. But there’s a project on <a href="https://github.com/Green-Software-Foundation/green-software-pattern">GitHub</a> working on this - let’s support it to make it as complete as possible.</p>
<h2 id="designing-sustainable-applications">Designing sustainable applications</h2>
<p>When it comes to climate change you might feel that as a software developer you don’t really have a lot of things you can do to save the planet.
That’ wrong - there is stuff we can do, the layered approach will guide you through the different areas. The bad news is however: As soon as you run your application it will consume energy and therefore impact climate change. So we can’t really write “green” software.</p>
<p>This leaves us with two options:</p>
<ol>
<li>We can try to build applications which are less bad.</li>
<li>We can come up with ideas how our software solutions can impact other areas to be more efficient and this way help reduce greenhouse gases.</li>
</ol>
<p>This post is addressing option 1 - at the same time I want to point out that I think the impact of option 2 can be huge. But not everyone of us is working on this kind of scenario and therefore I want to point out areas where you can make a difference by just focusing on the applications you are working on today.</p>
<h1 id="a-layered-model">A layered model</h1>
<p>When designing an application we have to make several choices before, during and after the development. A lot of these areas offer chances to make decisions which are more or less sustainable. I thought about this a lot and came up with this layered model which kind of describes my mental model around it.
This “layer thing” is easy to understand (as a person with software engineering background) and at the same time it helps to focus on specifc aspects at specific times and not always all layers are relevant.
As a disclaimer: I’m thinking about this from a cloud application point of view. So if you are more into mobile development, not all of the layers may be relevant to you (and there might be others).</p>
<p><img src="../images/2022-10-04-layers.jpg" alt="A stack of layers decribing different areas where decisions have to be made during software development" /></p>
<p>Let’s go through all layers one by one. For every layer I will point out some questions you can ask.</p>
<h3 id="ux-layer">UX Layer</h3>
<p>Questions to be asked:</p>
<ul>
<li>Will my users accept lower resolution images which might save bandwidth?</li>
<li>Is my provided service level accurate for the buisness scenario or can I lower it so that I can save resources?</li>
<li>Will my users accept a reduced performance by reducing compute power?
<h3 id="application-layer">Application Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>Can I switch to another framework or language which will be more efficient?
<h3 id="architecture-layer">Architecture Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>Can I scale in elastically to free up unused resources?</li>
<li>Can I increase the density of applications on a single host to improve utilization?</li>
<li>Can I design my application towards event-driven to avoid unnecessary polling?</li>
<li>Can I use cloud platform services where I share resources with other customers to increase utilization?</li>
<li>Can I use scale-out over scale-up to avoid higher clock speeds?
<h3 id="networking-layer">Networking Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>Can I reduce overall network traffic to save energy during data transfer?</li>
<li>Can I reduce the distance between client and service to reduce energy during data transfer?
<h3 id="data-layer">Data Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>Can I use caching to bring data closer to the user to reduce energy consumption during data transfer?</li>
<li>Can I use different database engines which work more efficently for specific types of queries?
<h3 id="physical-layer">Physical Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>Can I downsize my infrastructure components to improve utilization?</li>
<li>Can I move the location of my physical infrastructure towards a location which is running on green energy?</li>
<li>Can I time shift my computations towards a point in time where green energy is being used (e.g. solar power)?</li>
<li>Can I use a different chipset which will be more efficient for my task (e.g. ARM vs. X86)
<h3 id="governance-layer">Governance Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>In an enterprise can I cross charge cost for resource consumption so that people using resources will pay for it on a pay-as-you-go basis?</li>
<li>In an enterprise can I enforce de-provisioning of unused machines?
<h3 id="provider-layer">Provider Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>Can I chose a provider with a green footprint?
<h3 id="cultural-layer">Cultural Layer</h3>
<p>Questions to be asked:</p>
</li>
<li>How can I ensure sustainability in software design is a priority for the leadership team?</li>
<li>How can I ensure sustainability in software is a topic software developers care about?</li>
<li>Can I build communities around sustainability in software?</li>
</ul>
<h2 id="upsides-and-downsides">Upsides and downsides</h2>
<p>To be clear: All of those topics might impact other areas of your applications. They might touch on reliabilty, useability, customer satisfaction, ease of work, cost, and other. But the same goes for all other aspect we build in our applications. Application design always is based on trade-offs. Sustainability should just be part of the equation.
At the same time I did not touch on stuff like remote work and development processes - even though this might be very important here as well. Maybe I’ll add this in the future.</p>
<h2 id="resource-to-check">Resource to check</h2>
<p>If you are interested in this topic, definitly also check out the following websites which will help you learn more about the impact of software on climate change. It’s an interesting read and most of the stuff is open source and open for contributions.</p>
<p><a href="https://principles.green/">Principles Green</a></p>
<p><a href="https://patterns.greensoftware.foundation/">Patterns for sustainable Software</a></p>
<p><a href="https://devblogs.microsoft.com/sustainable-software/">Sustainable Software Engineering Blog</a></p>
<p><a href="https://greensoftware.foundation/">Green Software Foundation</a></p>
<p>If you read those sites you will find that all of this is somehow interconnected. To me this indicates the information out there in the internet is still very limited today. Let’s work on changing this!</p>tl;dr; There are multiple areas where we can adjust applications to reduce carbon emissions. Some of them might not be relevant to you, others might be out of your authority. It’s good to have an understanding that those areas exist - this is where I think the layered approach can help. At this point (October 2022) there’s no wholistic list of recommended patterns to build more sustainable applications. But there’s a project on GitHub working on this - let’s support it to make it as complete as possible.Retrieve secrets from Azure Keyvault into your GitHub Codespaces2022-01-04T00:00:00+00:002022-01-04T00:00:00+00:00https://danielmeixner.github.io//codespaces-secrets<p>During development at some point you might need to use a secret, e.g. a password or connection string either within your development environment or while debugging. For GitHub Codespaces a solution exists with <a href="https://docs.github.com/en/codespaces/managing-codespaces-for-your-organization/managing-encrypted-secrets-for-your-repository-and-organization-for-codespaces">Codespaces Secrets in GitHub </a> and this actually works pretty good.</p>
<p>I was looking for a way how to controll my secrets within an Azure KeyVault, e.g. to be able to share secrets with peers outside of my organisation and because I like the handling and API and GUI better in Azure KeyVault. So my idea was to integrate GitHub codespaces with Azure Keyvault.</p>
<p>I found the way described below and can now access Keyvault secrets via environment variables from both my commandline in Codespaces and from the app.</p>
<h2 id="heres-what-i-did">Here’s what I did:</h2>
<ol>
<li>Create an Azure KeyVault with some secrets.</li>
<li>Create a Service Principal on Azure Commandline.
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>az ad spn create-for-rbac -n GHCodespaces
</code></pre></div> </div>
</li>
<li>Create a GitHub Codespaces secret and paste the output from the previous command into it. Make sure you name the secret AZSPN.</li>
<li>Create another GitHub Codespaces secret and provide the name of your keyvault. Make sure you name this secret VAULTNAME.</li>
<li>Create an <a href="https://docs.microsoft.com/en-us/azure/key-vault/general/quick-create-portal">Azure Keyvault</a> and an an access policy in Azure Keyvault to give your Service Principal secret list and secret get permissions.</li>
<li>In your Codespace modify the .devcontainer.json file to call the script I’m providing <a href="https://github.com/DanielMeixner/demo-AzureSecrets/blob/main/.devcontainer/readSecretsFromKeyVaultToEnv.sh">here</a> as a poststartcommand.</li>
<li>Additionally set remoteuser and containeruser in .devcontainer.json to “root”.</li>
<li>You can find a full sample implementation in the repo provided <a href="https://github.com/DanielMeixner/demo-AzureSecrets/">https://github.com/DanielMeixner/demo-AzureSecrets/</a></li>
</ol>
<h2 id="this-is-how-it-works">This is how it works</h2>
<ul>
<li>Under the hood the script simply uses the Azure Service Principal to reach out to the Azure Keyvault. Therefore some string operations are required to get username, password, tenant from the GitHub secret in the first place. (You can see them in the script file)</li>
<li>Then for every secret found a new environment variable will be exported to the .bashrc file using the export command and bashrc will be reloaded.</li>
<li>The environment variable will be all upper case and - will be replaced with _</li>
</ul>
<h2 id="try-yourself">Try yourself</h2>
<ul>
<li>If you want to give it a try, just fork the <a href="https://github.com/DanielMeixner/demo-AzureSecrets/">demo repo</a>.</li>
<li>Then follow the steps above. Make sure you provide access to your GitHub Codespaces repo to your repo.</li>
<li>For demo purposes add a secret in your keyvault instance. Name the secret pw.</li>
<li>Open the demo in a Codespace</li>
<li>To check if everything worked, just run this on the commandline to start the demo app:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node index.js
</code></pre></div> </div>
<p>If everything worked, you should see that you get the secret value returned.</p>
</li>
</ul>
<blockquote>
<p><strong>Hint</strong></p>
<p>Never store secrets in files which are in danger to be committed to source control. In our case the secrets are stored in .bashrc which is not in the working tree per default. If you have an idea how to get it done without even writing to a file let me know.</p>
</blockquote>During development at some point you might need to use a secret, e.g. a password or connection string either within your development environment or while debugging. For GitHub Codespaces a solution exists with Codespaces Secrets in GitHub and this actually works pretty good.az ad sp create-for-rbac -n - Values of identifierUris …2022-01-03T00:00:00+00:002022-01-03T00:00:00+00:00https://danielmeixner.github.io//Az-ad-sp-create-for-rbac-Values%20of%20identifierUris%20property%20must%20use%20a%20verified%20domain<p>If you are running into the message</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Values of identifierUris property must use a verified domain of the organization or its subdomain
</code></pre></div></div>
<p>after executing “az ad sp create-for-rbac” in Azure CLI, try updating your Azure CLI version.
Versions starting with 2.25 should fix the issue. More on this <a href="https://github.com/Azure/azure-cli/issues/19892">here</a></p>
<p>To check your version run</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>az version
</code></pre></div></div>
<p>To upgrade your CLI just run</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>az upgrade
</code></pre></div></div>
<p>In case az upgrade doesn’t work for you, you are on a version too old, so you have to <a href="https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli">install the upgrade manually</a>.</p>
<p>If you’re on an new version there’s also a way to automatically stay up to date:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>az config set auto-upgrade.enable=yes
</code></pre></div></div>If you are running into the message Values of identifierUris property must use a verified domain of the organization or its subdomain after executing “az ad sp create-for-rbac” in Azure CLI, try updating your Azure CLI version. Versions starting with 2.25 should fix the issue. More on this hereSetup GitHub pages with Codespaces2021-12-31T00:00:00+00:002021-12-31T00:00:00+00:00https://danielmeixner.github.io//setup-ghpages-codespaces<p>Here’s a superquick way to setup GitHub pages for your account, if you haven’t done this yet.</p>
<ol>
<li>Just import a new repo from https://github.com/DanielMeixner/GitHubPagesSetup (don’t fork it!).
<img src="../images/20211231-import.jpg" alt="Screenshot of import dialog" /></li>
<li>Name the repo YOUR-USERNAME.github.io (in my case DanielMeixner.github.io)</li>
<li>Open the new repo in a Codespace.</li>
</ol>
<p>**That’s it. You will find your new blog on https://YOUR-USERNAME.github.io in a few minutes. **</p>
<p>In the background automation will trigger a setup of your GitHub page using devcontainer automation and you can start blogging right away.</p>
<blockquote>
<p>Hint:
In case you already have another repo setup in YOUR-USERNAME.github.io you have to adjust the url in the _config.yml file to your new url. It will follow the pattern https://YOUR-USERNAME.github.com/YOUR-REPO-NAME . In this case you also have to explicitly enable Pages in the settings for your repo.</p>
</blockquote>Here’s a superquick way to setup GitHub pages for your account, if you haven’t done this yet. Just import a new repo from https://github.com/DanielMeixner/GitHubPagesSetup (don’t fork it!). Name the repo YOUR-USERNAME.github.io (in my case DanielMeixner.github.io) Open the new repo in a Codespace.Connect to a virtual network with GitHub Codespaces2021-12-30T00:00:00+00:002021-12-30T00:00:00+00:00https://danielmeixner.github.io//codespaces-vpn-azure<p>If you are interested in working with Codespaces and want to access resources in a private network here’s a guide how you can achive this.</p>
<p>Actually there’s already <a href="https://docs.github.com/en/codespaces/developing-in-codespaces/connecting-to-a-private-network">some documentation</a> on this topic which refers to a GitHub repository with <a href="https://github.com/codespaces-contrib/codespaces-openvpn">even more docs</a> here which then again links to other docs on how to achieve this. My post contains the essence of all of that boiled down to what I found important for myself. I hope it is useful for you as well. For more information, please check the original posts. I also stripped down the code to be as compact as possible. The original solution mentioned above is a little more sophisticated.</p>
<p>I’ll start with the interesting part and for completeness will then also quickly show the boring stuff that has to be done, to figure out if it really works.</p>
<h1 id="goal">Goal</h1>
<p>The goal is to connect to a virtual network in Azure from a GitHub Codespace. The VNET in Azure shall not be publicly accessible. GitHub Codespaces run on shared infrastructure in a datacenter out of my control. I want to be able to communicate with resources inside my VNET in Azure from my Codespace.</p>
<h1 id="solution">Solution:</h1>
<p>I create a VPN connection to my VNET using OpenSSL. The connection will be established whenever my CodeSpace starts.</p>
<h1 id="requirements">Requirements</h1>
<p>I don’t want to stored secrets in my repo.</p>
<h1 id="step-by-step">Step by step:</h1>
<p>Here’s what I do step by step.</p>
<h2 id="setup-your-codespace">Setup your codespace</h2>
<ol>
<li>
<p><strong>Create a CodeSpace for your repo on GitHub and add a .devcontainer file</strong>. You can do this manually but I recommend you use the <a href="https://docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/setting-up-your-project-for-codespaces?langId=java#step-2:-add-a-dev-container-to-your-codespace-from-a-template">wizard</a>. Depending on the language of your project you can pick a container that fits. In my case it was a node.js container.</p>
</li>
<li><strong>Modify the DOCKERFILE to install OpenVPN tools</strong> With the previous command you got a DOCKERFILE. This will be used to build your container. You can edit it as you like. We need to install the OpenVPN client software so we add these lines:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RUN export DEBIAN_FRONTEND=noninteractive && apt-get update \
&& apt-get -y install --no-install-recommends openvpn \
</code></pre></div> </div>
</li>
<li>
<p><strong>Add a poststartcommand to connect</strong> To connect we need to call a few commands.
We put those commands in a separate file. I put it into a subfolder in the ./devcontainer folder. This folder has been created automatically when you followed the wizard in step 1. Let’s name the file “install-dev-tools”. I put it into a subfolder “.devcontainer/postcreate/install-dev-tools.sh”.</p>
<p>This file will be called whenever your container starts up. Add the following line to the file.</p>
<p><em>Content of ./.devcontainer/postcreate/install-dev-tools.sh</em></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #!/usr/bin/env bash
mkdir -p ./.ignore
echo ".ignore/" >> .gitignore
echo "${OPENVPNCONFIG}" > ./.ignore/openvpn.config
nohup sudo /bin/sh -c " openvpn --log ./.ignore/openvpn.log --config ./.ignore/openvpn.config &"
</code></pre></div> </div>
<p>To actually call the file we add this section to the devcontainer.json file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"postStartCommand": "bash ./.devcontainer/postcreate/install-dev-tools.sh",
</code></pre></div> </div>
<p><strong>What does it?</strong></p>
<ul>
<li>First it creates a .ignore directory. (mkdir …)</li>
<li>It adds the .ignore folder to the .gitignore file to make sure it will never be part of your repository and therefore not end up in the version history. (echo “.ignore/” » .gitignore ) <strong>This is important!</strong> Otherwise everybody could connect to your VPN, if you accidentally push this to a public repo.</li>
<li>It creates a file which contain the configuration for your vpn connection. This information is read from an environment variable. (echo “${OPENVPNCONFIG}” …)</li>
<li>Then it opens a VPN connection. We use nohup here to avoid hangups. We also run with sudo - this might or might not be neccessary in your case. And of course we refer to the newly created file for config. We log to the same folder. (nohup sudo openvpn…)</li>
</ul>
</li>
<li>
<p><strong>Create a GitHub Codespaces secret</strong> which contains the required information to create a VPN connection. We create a new secret as described <a href="https://docs.github.com/en/codespaces/managing-your-codespaces/managing-encrypted-secrets-for-your-codespaces#adding-a-secret">here</a>. Call this secret OPENVPNCONFIG. You could call it whatever you like but for this tutorial stick with this name.
You might wonder how you get the “required information”. Yes, that’s the tricky part. We are looking for an OpenSSL configuration here. We come to this in a second. For now you can create the secret. If you already have the info available, perfect. If not, revisit this step later and update your secret.</p>
</li>
<li><strong>Make the secret available to your repo</strong>. You have to explicitly allow CodeSpaces to read this secret. This is also described <a href="https://docs.github.com/en/codespaces/managing-your-codespaces/managing-encrypted-secrets-for-your-codespaces#adding-a-secret">here</a>. Make sure the Codespace you want to connect can read the secret. You have to restart the Codespace to be able to actually read it.</li>
</ol>
<p>This is it, basically. The tricky part might be how to get to the content of the secret. If you don’t know how to get this openvpn config, read the next chapter.</p>
<h2 id="create-the-openssl-config">Create the OpenSSL Config</h2>
<ol>
<li>First set up a Virtual network Gateway in Azure.</li>
<li>Set up a Point-to-Site connection for OpenSSL. This is described <a href="https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal">here</a>. It’s all straight forward, the tricky part is the creation of certs. You will end up with a Virtual Network on Azure and a VPN gateway if you follwo this tutorial. When creating the Point-to-Site connection choose tunnel tpye OpenVPN.</li>
<li>Create the certificates. This is described <a href="https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-certificates-point-to-site">here</a> as well. It all boils down to a few PowerShell commands.</li>
<li>Export the certificates (both root and client cert) as described <a href="https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-certificates-point-to-site#cer">here</a> You will end up with a .cer file and a .pfx file.</li>
<li>Hint: You will need the server cert to be able to set up the point to site connection. For this you can simply open your .cer file in a texteditor and copy the content into the “Public certifacte data” field in Azure Portal. Do not copy the “—-BEGIN CERTIFICATE …” and “—–END …” lines, just the content.</li>
<li>If you are finished hit the “Download VPN client” button in Azure Portal on your Point-to-Site configuration.
<img src="images/20211230vpn.jpg" alt="Azure Portal" /></li>
</ol>
<p>You will get a bunch of stuff but will also find a OpenVPN folder with an vpnconfig.ovpn file. Open the file in a text editor. You will see, there are two sections missing. You have to replace $CLIENTCERTIFICATE and $PRIVATEKEY. Run this command to get the values.
<code class="language-plaintext highlighter-rouge">
openssl pkcs12 -in clientcert.pfx -out clientcertvalues.pem -nodes
</code>
You will get a clientcertvalues.pem file as a result. You can open this in a text editor and will find the client certificate and the private key in there. <strong>Do NOT put this file into source control!</strong> A private key is called private key because it is meant to stay private! Don’t share it!</p>
<ol>
<li>Replace the values in vpnconfig.ovpn as described. No copy the content of this file to the clipboard and paste it as a value into the GitHub Codespaces secret you already created. You might realize that you can not access the content of your existing secret. That’s on purpose, but you can <a href="https://docs.github.com/en/codespaces/managing-your-codespaces/managing-encrypted-secrets-for-your-codespaces#editing-a-secret">edit</a> the value of the secret. Just replace it with the content of your clipboard. Didn’t I just say you should not share this secret? Yes! You can still share it as a secret in GitHub because you will be the only one allowed to access this value. It will not be visible in your repository, even if you make the repo public.</li>
</ol>
<p>That’s it. Now let’s test if it really works.</p>
<h2 id="test-it">Test it</h2>
<ol>
<li>Add a virtual machine to the Virtual Network you just created. I chose a ubuntu server.</li>
<li>Open port 8080 in the network configuration.</li>
<li>Log into this VM via ssh.</li>
<li>Install ruby into the VM.
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install ruby
</code></pre></div> </div>
</li>
<li>Create a simple HTML page in a www folder.
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir -p ./www
echo "<HTML><BODY>Hello Codespaces </HTML></BODY>" > ./www/index.html
cd ./www
</code></pre></div> </div>
</li>
<li>Run a webserver to deliver this website.
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ruby -run -e httpd . -p 8080
</code></pre></div> </div>
</li>
<li>Now you have a webserver running. Keep it running.</li>
<li>
<p>In the Azure portal check the private IP your machine got. (Not the public one!)</p>
</li>
<li>Now restart your Codespace. On the commandline run
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl YOUR-PRIVATE-IP:8080
</code></pre></div> </div>
<p>You should see a nice greeting from your ruby webserver.
Don’t forget to delete the server afterwards.</p>
</li>
</ol>
<p>Please make sure you read the following docs for the full picture:</p>
<ul>
<li><a href="https://docs.github.com/en/codespaces/developing-in-codespaces/connecting-to-a-private-network">Official docs on VPN connectivity from Codespaces </a></li>
<li><a href="https://github.com/codespaces-contrib/codespaces-openvpn#:~:text=Using%20the%20OpenVPN%20client%20from%20GitHub%20Codespaces%20GitHub,machine%20or%20the%20network%20it%20is%20sitting%20in.">GitHub repo on VPN connectivity from Codespaces</a></li>
<li><a href="https://code.visualstudio.com/docs/remote/create-dev-container">Info on DevContainer</a></li>
</ul>If you are interested in working with Codespaces and want to access resources in a private network here’s a guide how you can achive this.